Jul 27 19:39:45.319758 ip-10-0-142-140 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 19:39:45.762090 ip-10-0-142-140 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 19:39:45.762090 ip-10-0-142-140 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 19:39:45.762090 ip-10-0-142-140 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 19:39:45.762090 ip-10-0-142-140 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 19:39:45.762090 ip-10-0-142-140 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 19:39:45.763254 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.763136    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 19:39:45.768953 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.768930    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:39:45.768953 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.768948    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:39:45.768953 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.768953    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:39:45.768953 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.768957    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:39:45.768953 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.768961    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.768965    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.768969    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.768974    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.768978    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.768982    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.768986    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.768990    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.768994    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.768998    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769009    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769014    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769018    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769022    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769025    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769030    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769034    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769039    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769044    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769047    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:39:45.769265 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769051    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769055    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769059    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769063    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769067    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769071    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769075    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769082    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769087    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769091    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769095    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769099    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769103    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769108    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769114    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769119    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769123    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769127    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769132    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769137    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:39:45.770084 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769141    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769145    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769150    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769154    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769159    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769163    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769168    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769172    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769178    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769183    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769190    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769197    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769202    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769207    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769212    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769216    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769220    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769225    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769230    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769235    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:39:45.770979 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769239    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769244    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769248    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769252    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769257    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769261    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769265    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769269    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769274    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769278    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769282    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769286    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769290    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769295    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769304    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769309    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769314    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769319    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769324    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:39:45.771820 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769329    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769334    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769338    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769343    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769348    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769353    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769357    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769397    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769405    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769410    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769415    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769421    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769426    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769430    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769435    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.769440    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770655    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770666    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770671    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770675    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:39:45.772645 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770679    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770684    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770688    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770692    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770696    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770702    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770707    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770711    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770715    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770719    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770724    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770728    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770732    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770737    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770741    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770746    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770752    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770759    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770763    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770767    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:39:45.773508 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770772    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770777    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770781    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770788    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770793    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770797    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770802    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770806    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770810    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770815    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770819    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770823    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770829    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770833    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770837    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770841    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770846    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770850    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770854    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770859    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:39:45.774222 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770863    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770868    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770873    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770877    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770881    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770885    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770890    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770894    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770898    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770902    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770907    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770912    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770916    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770923    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770929    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770935    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770939    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770944    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770949    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:39:45.774733 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770953    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770957    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770962    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770966    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770970    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770974    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770978    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770982    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770986    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770990    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770993    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.770998    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771002    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771006    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771010    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771015    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771020    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771024    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771028    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771033    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:39:45.775264 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771037    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771042    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771046    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771050    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771055    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771060    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771065    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771069    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771073    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771077    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771082    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771089    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771094    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771100    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771104    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771109    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771773    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771782    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771787    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:39:45.775905 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771791    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771796    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771800    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771804    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771809    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771813    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771817    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771821    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771825    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771829    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771833    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771849    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771854    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771858    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771866    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771872    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771877    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771890    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771895    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:39:45.776548 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771899    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771904    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771908    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771912    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771916    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771921    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771925    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771929    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771934    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771938    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771942    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771947    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771951    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771958    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771963    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771968    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771973    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771977    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771982    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:39:45.777045 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771986    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771990    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771994    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.771998    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772002    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772007    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772011    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772015    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772019    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772026    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772030    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772034    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772046    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772050    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772054    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772058    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772062    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772067    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772074    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772078    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:39:45.777564 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772083    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772087    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772091    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772094    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772099    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772103    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772108    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772112    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772116    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772120    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772124    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772128    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772132    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772136    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772141    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772145    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772149    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772153    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772158    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772163    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:39:45.778104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772168    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772172    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772176    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772180    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772184    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772196    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772200    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772204    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772208    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772212    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772217    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772221    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772225    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772229    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772233    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772238    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772244    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.772248    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772351    2562 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772362    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772381    2562 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 19:39:45.778650 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772393    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772399    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772404    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772410    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772417    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772423    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772428    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772433    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772438    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772443    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772448    2562 flags.go:64] FLAG: --cgroup-root=""
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772452    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772457    2562 flags.go:64] FLAG: --client-ca-file=""
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772461    2562 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772466    2562 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772497    2562 flags.go:64] FLAG: --cluster-domain=""
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772501    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772514    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772520    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772525    2562 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772531    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772537    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772542    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 19:39:45.779161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772547    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772552    2562 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772557    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772561    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772569    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772574    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772581    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772586    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772591    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772595    2562 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772601    2562 flags.go:64] FLAG: --enable-server="true"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772605    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772617    2562 flags.go:64] FLAG: --event-burst="100"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772622    2562 flags.go:64] FLAG: --event-qps="50"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772627    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772632    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772636    2562 flags.go:64] FLAG: --eviction-hard=""
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772642    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772647    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772651    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772656    2562 flags.go:64] FLAG: --eviction-soft=""
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772660    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772664    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772669    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772674    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 19:39:45.779774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772679    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772683    2562 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772696    2562 flags.go:64] FLAG: --feature-gates=""
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772702    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772707    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772712    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772717    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772722    2562 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772727    2562 flags.go:64] FLAG: --help="false"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772732    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-142-140.ec2.internal"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772737    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772743    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772748    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772754    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772759    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772764    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772770    2562 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772774    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772779    2562 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772784    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772789    2562 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772793    2562 flags.go:64] FLAG: --kube-reserved=""
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772798    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772802    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 19:39:45.780393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772807    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772812    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772817    2562 flags.go:64] FLAG: --lock-file=""
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772822    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772827    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772831    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772840    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772847    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772852    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772857    2562 flags.go:64] FLAG: --logging-format="text"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772861    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772875    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772880    2562 flags.go:64] FLAG: --manifest-url=""
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772884    2562 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772892    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772896    2562 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772903    2562 flags.go:64] FLAG: --max-pods="110"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772908    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772913    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772919    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772924    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772928    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772934    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772938    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772955    2562 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 19:39:45.781304 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772960    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772965    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772970    2562 flags.go:64] FLAG: --pod-cidr=""
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772974    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772986    2562 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772991    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.772996    2562 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773010    2562 flags.go:64] FLAG: --port="10250"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773015    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773020    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05e99e58768e6559b"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773025    2562 flags.go:64] FLAG: --qos-reserved=""
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773030    2562 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773035    2562 flags.go:64] FLAG: --register-node="true"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773039    2562 flags.go:64] FLAG: --register-with-taints=""
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773045    2562 flags.go:64] FLAG: --registry-burst="10"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773052    2562 flags.go:64] FLAG: --registry-qps="5"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773057    2562 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773062    2562 flags.go:64] FLAG: --reserved-memory=""
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773068    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773080    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773085    2562 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773090    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773095    2562 flags.go:64] FLAG: --runonce="false"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773099    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773105    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 19:39:45.782251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773110    2562 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773116    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773120    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773126    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773130    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773136    2562 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773140    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773145    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773150    2562 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773154    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773159    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773164    2562 flags.go:64] FLAG: --system-cgroups=""
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773168    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773177    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773181    2562 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773186    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773196    2562 flags.go:64] FLAG: --tls-min-version=""
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773201    2562 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773205    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773210    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773214    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773219    2562 flags.go:64] FLAG: --v="2"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773227    2562 flags.go:64] FLAG: --version="false"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773234    2562 flags.go:64] FLAG: --vmodule=""
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773240    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 19:39:45.783165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.773245    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773470    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773505    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773510    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773515    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773519    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773524    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773528    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773534    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773538    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773542    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773546    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773550    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773554    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773559    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773562    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773567    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773571    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773575    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773579    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:39:45.783787 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773583    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773587    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773591    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773596    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773600    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773604    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773608    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773613    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773616    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773622    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773626    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773630    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773634    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773641    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773647    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773659    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773664    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773668    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773673    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:39:45.784309 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773677    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773683    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773687    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773691    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773695    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773699    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773703    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773707    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773711    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773715    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773719    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773724    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773728    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773732    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773736    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773740    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773744    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773748    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773752    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773756    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:39:45.784800 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773760    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773764    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773771    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773775    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773780    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773784    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773788    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773792    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773795    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773807    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773812    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773816    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773820    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773826    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773830    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773834    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773838    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773842    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773846    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:39:45.785290 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773853    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773859    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773863    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773868    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773873    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773877    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773882    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773894    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773899    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773903    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773907    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773912    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773917    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773923    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773928    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773933    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773937    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773941    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773945    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:39:45.785819 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773950    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:39:45.786277 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773954    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:39:45.786277 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.773958    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:39:45.786277 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.774756    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 19:39:45.786277 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.784155    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 19:39:45.786277 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.784172    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 19:39:45.786277 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.784193    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 19:39:45.786277 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.784199    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 19:39:45.786277 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784256    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:39:45.786277 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784261    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:39:45.786277 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784264    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:39:45.786277 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784267    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:39:45.786277 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784271    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:39:45.786277 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784275    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:39:45.786277 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784279    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784282    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784284    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784287    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784289    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784292    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784295    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784297    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784300    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784303    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784306    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784309    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784311    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784314    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784317    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784320    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784322    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784325    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784328    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784330    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:39:45.786625 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784332    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784336    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784340    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784344    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784347    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784349    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784352    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784354    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784357    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784359    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784362    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784364    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784367    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784369    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784372    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784374    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784377    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784379    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784382    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:39:45.787099 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784385    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784388    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784392    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784395    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784398    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784401    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784403    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784406    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784409    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784411    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784414    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784417    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784420    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784422    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784425    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784427    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784431    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784433    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784436    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:39:45.787573 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784438    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784441    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784443    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784446    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784448    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784451    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784453    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784456    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784458    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784461    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784463    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784465    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784468    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784471    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784473    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784505    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784507    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784509    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784512    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784514    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784517    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:39:45.788039 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784520    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784522    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784525    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784527    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784530    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784533    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784535    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784538    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784541    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784544    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784546    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784549    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784551    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784554    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.784559    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 19:39:45.788556 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784695    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784701    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784705    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784707    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784711    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784715    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784718    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784721    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784724    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784727    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784730    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784732    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784736    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784739    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784742    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784744    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784747    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784749    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784752    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784755    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:39:45.788922 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784757    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784760    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784762    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784765    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784767    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784770    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784773    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784775    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784778    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784780    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784783    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784785    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784788    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784791    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784793    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784795    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784798    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784800    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784802    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784805    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:39:45.789410 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784808    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784811    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784813    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784816    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784819    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784822    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784824    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784827    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784829    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784831    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784834    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784836    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784839    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784843    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784846    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784849    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784852    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784854    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784857    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:39:45.789920 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784859    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784862    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784865    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784869    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784871    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784874    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784876    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784878    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784881    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784883    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784886    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784889    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784891    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784894    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784896    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784898    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784901    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784903    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784906    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:39:45.790364 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784909    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784911    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784914    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784916    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784918    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784921    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784923    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784926    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784928    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784931    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784933    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784936    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784938    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784940    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784943    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784945    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784947    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784950    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784952    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784955    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:39:45.790839 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:45.784958    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:39:45.791315 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.784962    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 19:39:45.791315 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.785078    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 19:39:45.791315 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.790075    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 19:39:45.791783 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.791771    2562 server.go:1019] "Starting client certificate rotation"
Jul 27 19:39:45.791888 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.791872    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 19:39:45.791923 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.791904    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 19:39:45.794774 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.794762    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 19:39:45.798806 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.798792    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 19:39:45.811078 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.811061    2562 log.go:25] "Validated CRI v1 runtime API"
Jul 27 19:39:45.815247 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.815230    2562 log.go:25] "Validated CRI v1 image API"
Jul 27 19:39:45.817090 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.817074    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 19:39:45.822030 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.822009    2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b582524e-62b6-48f2-b3ea-45b95f848f7e:/dev/nvme0n1p3 d8b8c33d-2cfb-46f4-9371-6b5b09984ade:/dev/nvme0n1p4]
Jul 27 19:39:45.822100 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.822029    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 19:39:45.823169 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.823146    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 19:39:45.827116 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.827004    2562 manager.go:216] Machine: {Timestamp:2026-07-27 19:39:45.82560745 +0000 UTC m=+0.377523859 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095468 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2975cafa9076dc29d5c618b2f34787 SystemUUID:ec2975ca-fa90-76dc-29d5-c618b2f34787 BootID:dd8d6b38-f990-4458-82bd-89adbc59de63 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9c:d3:66:18:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9c:d3:66:18:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:0c:43:6a:93:cc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 19:39:45.827116 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.827110    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 19:39:45.827232 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.827220    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 19:39:45.830089 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.830068    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 19:39:45.830221 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.830091    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-140.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 19:39:45.830934 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.830924    2562 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 19:39:45.830969 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.830936    2562 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 19:39:45.830969 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.830944    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 19:39:45.831797 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.831787    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 19:39:45.833166 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.833155    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 19:39:45.835987 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.835976    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 19:39:45.836092 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.836083    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 19:39:45.836133 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.836112    2562 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 19:39:45.836133 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.836121    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 19:39:45.836202 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.836136    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 19:39:45.836202 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.836145    2562 kubelet.go:398] "Adding apiserver pod source"
Jul 27 19:39:45.836202 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.836154    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 19:39:45.836649 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.836628    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 19:39:45.836733 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.836665    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 19:39:45.846840 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.846814    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 19:39:45.849201 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.849186    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 19:39:45.849273 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.849217    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 19:39:45.851184 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.851171    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 19:39:45.851229 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.851188    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 19:39:45.851229 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.851194    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 19:39:45.851229 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.851199    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 19:39:45.851229 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.851208    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 19:39:45.851229 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.851213    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 19:39:45.851229 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.851220    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 19:39:45.851229 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.851226    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 19:39:45.851229 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.851232    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 19:39:45.851431 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.851237    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 19:39:45.851431 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.851246    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 19:39:45.851431 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.851262    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 19:39:45.852893 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.852879    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 19:39:45.852893 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.852893    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 19:39:45.854368 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:45.854338    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 19:39:45.854368 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:45.854360    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 19:39:45.856186 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.856171    2562 server.go:1268] "Started kubelet"
Jul 27 19:39:45.856277 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.856245    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 19:39:45.856277 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.856243    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 19:39:45.856407 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.856305    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 19:39:45.857038 ip-10-0-142-140 systemd[1]: Started Kubernetes Kubelet.
Jul 27 19:39:45.858442 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.858429    2562 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 19:39:45.860944 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.860926    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 19:39:45.861810 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:45.861792    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 19:39:45.864580 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.864564    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-140.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 19:39:45.865856 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:45.864739    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-140.ec2.internal.18c63cd48aaa7bf2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-140.ec2.internal,UID:ip-10-0-142-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-140.ec2.internal,},FirstTimestamp:2026-07-27 19:39:45.856142322 +0000 UTC m=+0.408058730,LastTimestamp:2026-07-27 19:39:45.856142322 +0000 UTC m=+0.408058730,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-140.ec2.internal,}"
Jul 27 19:39:45.866544 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.866522    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nfsrx"
Jul 27 19:39:45.870265 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.870245    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 19:39:45.870349 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.870250    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 19:39:45.870349 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:45.870326    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found"
Jul 27 19:39:45.871721 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.871707    2562 factory.go:55] Registering systemd factory
Jul 27 19:39:45.871837 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.871828    2562 factory.go:223] Registration of the systemd container factory successfully
Jul 27 19:39:45.872453 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:45.872432    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 19:39:45.872577 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:45.872550    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 19:39:45.872647 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.872608    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 19:39:45.872647 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.872626    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 19:39:45.872744 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.872706    2562 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 19:39:45.872744 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.872716    2562 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 19:39:45.873181 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.872873    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 19:39:45.873181 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.873159    2562 factory.go:153] Registering CRI-O factory
Jul 27 19:39:45.873181 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.873174    2562 factory.go:223] Registration of the crio container factory successfully
Jul 27 19:39:45.873342 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.873225    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 19:39:45.873342 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.873248    2562 factory.go:103] Registering Raw factory
Jul 27 19:39:45.873342 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.873262    2562 manager.go:1196] Started watching for new ooms in manager
Jul 27 19:39:45.873726 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.873714    2562 manager.go:349] Starting recovery of all containers
Jul 27 19:39:45.876047 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.875982    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nfsrx"
Jul 27 19:39:45.884643 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.884627    2562 manager.go:354] Recovery completed
Jul 27 19:39:45.888595 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.888582    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:39:45.890294 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.890276    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:39:45.890355 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.890309    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:39:45.890355 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.890323    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:39:45.890901 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.890889    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 19:39:45.890978 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.890901    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 19:39:45.890978 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.890906    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:39:45.890978 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.890918    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 19:39:45.891626 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.891607    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:39:45.891678 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.891637    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:39:45.891678 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.891646    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:39:45.893661 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.893648    2562 policy_none.go:49] "None policy: Start"
Jul 27 19:39:45.893661 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.893663    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 19:39:45.893744 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.893672    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 19:39:45.895567 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.895558    2562 policy_none.go:47] "Start"
Jul 27 19:39:45.932351 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.932173    2562 manager.go:337] "Starting Device Plugin manager"
Jul 27 19:39:45.932449 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:45.932388    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 19:39:45.932449 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.932401    2562 server.go:87] "Starting device plugin registration server"
Jul 27 19:39:45.932662 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.932651    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 19:39:45.932709 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.932667    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 19:39:45.932870 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.932856    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 19:39:45.932943 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.932933    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 19:39:45.932987 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.932945    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 19:39:45.933435 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:45.933413    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 19:39:45.933533 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:45.933471    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-140.ec2.internal\" not found"
Jul 27 19:39:45.983136 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.983106    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 19:39:45.985350 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.984420    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 19:39:45.985350 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.984451    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 19:39:45.985350 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.984472    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 19:39:45.985350 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:45.984517    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 19:39:45.987453 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:45.987435    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:39:46.033079 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.033019    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:39:46.034291 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.034276    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:39:46.034364 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.034305    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:39:46.034364 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.034315    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:39:46.034364 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.034336    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.047399 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.047379    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.047507 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:46.047400    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-140.ec2.internal\": node \"ip-10-0-142-140.ec2.internal\" not found"
Jul 27 19:39:46.068093 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:46.068071    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found"
Jul 27 19:39:46.085133 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.085112    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal"]
Jul 27 19:39:46.085212 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.085188    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:39:46.086674 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.086659    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:39:46.086746 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.086685    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:39:46.086746 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.086694    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:39:46.088957 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.088945    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:39:46.089077 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.089061    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.089121 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.089092    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:39:46.089859 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.089843    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:39:46.089932 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.089870    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:39:46.089932 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.089881    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:39:46.089932 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.089843    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:39:46.090067 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.089957    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:39:46.090067 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.089976    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:39:46.092055 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.092040    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.092114 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.092067    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:39:46.092739 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.092725    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:39:46.092825 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.092750    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:39:46.092825 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.092763    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:39:46.121080 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:46.121055    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-140.ec2.internal\" not found" node="ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.124440 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:46.124422    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-140.ec2.internal\" not found" node="ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.168448 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:46.168416    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found"
Jul 27 19:39:46.174875 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.174852    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/028ba8eadc770529daa4c0f4fe338d90-config\") pod \"kube-apiserver-proxy-ip-10-0-142-140.ec2.internal\" (UID: \"028ba8eadc770529daa4c0f4fe338d90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.174952 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.174881    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/580850b6194cd2dae03e2367ddc94987-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal\" (UID: \"580850b6194cd2dae03e2367ddc94987\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.174952 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.174900    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580850b6194cd2dae03e2367ddc94987-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal\" (UID: \"580850b6194cd2dae03e2367ddc94987\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.269099 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:46.269064    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found"
Jul 27 19:39:46.275580 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.275555    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/028ba8eadc770529daa4c0f4fe338d90-config\") pod \"kube-apiserver-proxy-ip-10-0-142-140.ec2.internal\" (UID: \"028ba8eadc770529daa4c0f4fe338d90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.275678 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.275590    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/580850b6194cd2dae03e2367ddc94987-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal\" (UID: \"580850b6194cd2dae03e2367ddc94987\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.275678 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.275614    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580850b6194cd2dae03e2367ddc94987-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal\" (UID: \"580850b6194cd2dae03e2367ddc94987\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.275765 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.275671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/580850b6194cd2dae03e2367ddc94987-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal\" (UID: \"580850b6194cd2dae03e2367ddc94987\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.275765 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.275677    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/028ba8eadc770529daa4c0f4fe338d90-config\") pod \"kube-apiserver-proxy-ip-10-0-142-140.ec2.internal\" (UID: \"028ba8eadc770529daa4c0f4fe338d90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.275765 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.275673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580850b6194cd2dae03e2367ddc94987-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal\" (UID: \"580850b6194cd2dae03e2367ddc94987\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.369830 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:46.369752    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found"
Jul 27 19:39:46.423807 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.423779    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.427676 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.427661    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.470235 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:46.470201    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found"
Jul 27 19:39:46.570760 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:46.570731    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found"
Jul 27 19:39:46.671298 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:46.671209    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found"
Jul 27 19:39:46.771849 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:46.771814    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-140.ec2.internal\" not found"
Jul 27 19:39:46.791294 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.791271    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 19:39:46.791462 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.791420    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 19:39:46.851692 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.851668    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:39:46.870649 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.870623    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.870649 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.870645    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 19:39:46.878915 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.878896    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 19:39:46.883408 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.883395    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 19:39:46.884444 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.884414    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 19:34:46 +0000 UTC" deadline="2028-03-05 15:40:29.911057353 +0000 UTC"
Jul 27 19:39:46.884507 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.884446    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14084h0m43.026614045s"
Jul 27 19:39:46.884873 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.884860    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal"
Jul 27 19:39:46.893841 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.893828    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 19:39:46.904306 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.904290    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vmlq4"
Jul 27 19:39:46.911227 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.911208    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vmlq4"
Jul 27 19:39:46.943047 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:46.943013    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028ba8eadc770529daa4c0f4fe338d90.slice/crio-3494b2a38ceb666e58c91f6795d2ebd9a37627592f80653c487351484a9ecaf5 WatchSource:0}: Error finding container 3494b2a38ceb666e58c91f6795d2ebd9a37627592f80653c487351484a9ecaf5: Status 404 returned error can't find the container with id 3494b2a38ceb666e58c91f6795d2ebd9a37627592f80653c487351484a9ecaf5
Jul 27 19:39:46.943272 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:46.943250    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580850b6194cd2dae03e2367ddc94987.slice/crio-d54d46e676e24fd93c60492a53ba11df6a5179e819d99140d3879221eb618ef6 WatchSource:0}: Error finding container d54d46e676e24fd93c60492a53ba11df6a5179e819d99140d3879221eb618ef6: Status 404 returned error can't find the container with id d54d46e676e24fd93c60492a53ba11df6a5179e819d99140d3879221eb618ef6
Jul 27 19:39:46.949628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.949611    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:39:46.989293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.989249    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" event={"ID":"580850b6194cd2dae03e2367ddc94987","Type":"ContainerStarted","Data":"d54d46e676e24fd93c60492a53ba11df6a5179e819d99140d3879221eb618ef6"}
Jul 27 19:39:46.990269 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:46.990248    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal" event={"ID":"028ba8eadc770529daa4c0f4fe338d90","Type":"ContainerStarted","Data":"3494b2a38ceb666e58c91f6795d2ebd9a37627592f80653c487351484a9ecaf5"}
Jul 27 19:39:47.148917 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.148888    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:39:47.205833 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.205760    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:39:47.837596 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.837557    2562 apiserver.go:52] "Watching apiserver"
Jul 27 19:39:47.846280 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.846246    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 19:39:47.846771 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.846748    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-8q4hl","openshift-network-operator/iptables-alerter-xh555","openshift-ovn-kubernetes/ovnkube-node-8bnpc","kube-system/global-pull-secret-syncer-jbl6q","kube-system/konnectivity-agent-x5d7t","kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf","openshift-cluster-node-tuning-operator/tuned-gm5lr","openshift-dns/node-resolver-bglxw","openshift-image-registry/node-ca-8qf2p","openshift-multus/multus-additional-cni-plugins-7mwwb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal","openshift-multus/multus-42l2f","openshift-multus/network-metrics-daemon-z2n28"]
Jul 27 19:39:47.849574 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.849551    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8qf2p"
Jul 27 19:39:47.852013 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.851977    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:47.852113 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:47.852058    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:39:47.852517 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.852495    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 19:39:47.852792 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.852774    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 19:39:47.853127 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.853107    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x9fcl\""
Jul 27 19:39:47.853345 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.853325    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 19:39:47.854274 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.854249    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.856307 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.856286    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 19:39:47.856530 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.856495    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sxrpj\""
Jul 27 19:39:47.856805 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.856783    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 19:39:47.856805 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.856803    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.857000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.856978    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 19:39:47.857334 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.857315    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 19:39:47.859873 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.859354    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 19:39:47.859873 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.859390    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 19:39:47.859873 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.859586    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x5d7t"
Jul 27 19:39:47.859873 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.859728    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b7t4v\""
Jul 27 19:39:47.861528 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.861508    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:47.861628 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:47.861596    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:39:47.862056 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.861895    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 19:39:47.862056 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.861914    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 19:39:47.862056 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.862021    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l7tx2\""
Jul 27 19:39:47.864068 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.864046    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.866267 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.866245    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 19:39:47.866510 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.866488    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hl6mg\""
Jul 27 19:39:47.866712 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.866697    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 19:39:47.866892 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.866876    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 19:39:47.867976 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.867677    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.870137 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.870104    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 19:39:47.870236 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.870148    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 19:39:47.870683 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.870660    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cqw8x\""
Jul 27 19:39:47.873148 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.873082    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bglxw"
Jul 27 19:39:47.875166 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.875147    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 19:39:47.875967 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.875945    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fn5xb\""
Jul 27 19:39:47.876182 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.876162    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 19:39:47.876408 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.876392    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:39:47.876518 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:47.876472    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:39:47.879040 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.879020    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.881082 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.881053    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xh555"
Jul 27 19:39:47.881847 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.881825    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 19:39:47.881948 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.881868    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 19:39:47.882093 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.882075    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 19:39:47.882412 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.882395    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 19:39:47.882483 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.882455    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 19:39:47.882646 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.882629    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 19:39:47.882713 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.882697    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tqzq6\""
Jul 27 19:39:47.883248 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.883229    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 19:39:47.883717 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.883697    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 19:39:47.883937 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.883918    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mphh9\""
Jul 27 19:39:47.884148 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.884131    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 19:39:47.884749 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.884725    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.884834 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.884763    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-socket-dir\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.884834 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.884794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-var-lib-cni-bin\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.884834 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.884819    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d50693b0-f760-4b28-8b31-514885f144ab-serviceca\") pod \"node-ca-8qf2p\" (UID: \"d50693b0-f760-4b28-8b31-514885f144ab\") " pod="openshift-image-registry/node-ca-8qf2p"
Jul 27 19:39:47.884995 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.884847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjb7x\" (UniqueName: \"kubernetes.io/projected/d50693b0-f760-4b28-8b31-514885f144ab-kube-api-access-sjb7x\") pod \"node-ca-8qf2p\" (UID: \"d50693b0-f760-4b28-8b31-514885f144ab\") " pod="openshift-image-registry/node-ca-8qf2p"
Jul 27 19:39:47.884995 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.884875    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwlcb\" (UniqueName: \"kubernetes.io/projected/2d58052c-f824-4f53-8459-6c6f16e4bac9-kube-api-access-dwlcb\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.884995 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.884903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00b00751-0c4b-431b-8699-6aad21590c67-hosts-file\") pod \"node-resolver-bglxw\" (UID: \"00b00751-0c4b-431b-8699-6aad21590c67\") " pod="openshift-dns/node-resolver-bglxw"
Jul 27 19:39:47.884995 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.884930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8413e456-000f-457a-9ad6-a337819268cc-cnibin\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.884995 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.884959    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8413e456-000f-457a-9ad6-a337819268cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.884995 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.884985    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-run-netns\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.885293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885008    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-hostroot\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.885293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885030    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-lib-modules\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.885293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885054    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-host\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.885293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8413e456-000f-457a-9ad6-a337819268cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.885293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-multus-conf-dir\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.885293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885122    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63bd477d-eac4-4ca6-b5d3-4734ba278992-multus-daemon-config\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.885293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885138    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc74q\" (UniqueName: \"kubernetes.io/projected/cb8b05dc-fed1-4ced-b993-489c5da47ecf-kube-api-access-fc74q\") pod \"network-metrics-daemon-z2n28\" (UID: \"cb8b05dc-fed1-4ced-b993-489c5da47ecf\") " pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:47.885293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvcxg\" (UniqueName: \"kubernetes.io/projected/8413e456-000f-457a-9ad6-a337819268cc-kube-api-access-wvcxg\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.885293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885175    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63bd477d-eac4-4ca6-b5d3-4734ba278992-cni-binary-copy\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.885293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885190    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-sysctl-conf\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.885293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885207    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/003428df-67c2-4b3c-906a-260c8728b23a-kubelet-config\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:47.885293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885234    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-kubernetes\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.885293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885259    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-sysctl-d\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.885293 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vtk2\" (UniqueName: \"kubernetes.io/projected/63bd477d-eac4-4ca6-b5d3-4734ba278992-kube-api-access-7vtk2\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885305    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4btjr\" (UniqueName: \"kubernetes.io/projected/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-kube-api-access-4btjr\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8413e456-000f-457a-9ad6-a337819268cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-etc-kubernetes\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885350    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-tuned\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885368    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/003428df-67c2-4b3c-906a-260c8728b23a-dbus\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885387    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00b00751-0c4b-431b-8699-6aad21590c67-tmp-dir\") pod \"node-resolver-bglxw\" (UID: \"00b00751-0c4b-431b-8699-6aad21590c67\") " pod="openshift-dns/node-resolver-bglxw"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-var-lib-kubelet\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-run\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885454    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885469    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-modprobe-d\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885495    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d50693b0-f760-4b28-8b31-514885f144ab-host\") pod \"node-ca-8qf2p\" (UID: \"d50693b0-f760-4b28-8b31-514885f144ab\") " pod="openshift-image-registry/node-ca-8qf2p"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-etc-selinux\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-multus-cni-dir\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-cnibin\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-os-release\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-sysconfig\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.886000 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885640    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af41025b-248c-4634-8c46-68d227fa3478-konnectivity-ca\") pod \"konnectivity-agent-x5d7t\" (UID: \"af41025b-248c-4634-8c46-68d227fa3478\") " pod="kube-system/konnectivity-agent-x5d7t"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885654    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-registration-dir\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885669    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-device-dir\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885683    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-var-lib-kubelet\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885701    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs\") pod \"network-metrics-daemon-z2n28\" (UID: \"cb8b05dc-fed1-4ced-b993-489c5da47ecf\") " pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8413e456-000f-457a-9ad6-a337819268cc-system-cni-dir\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshb5\" (UniqueName: \"kubernetes.io/projected/00b00751-0c4b-431b-8699-6aad21590c67-kube-api-access-lshb5\") pod \"node-resolver-bglxw\" (UID: \"00b00751-0c4b-431b-8699-6aad21590c67\") " pod="openshift-dns/node-resolver-bglxw"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-var-lib-cni-multus\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885783    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8413e456-000f-457a-9ad6-a337819268cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885801    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-multus-socket-dir-parent\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-run-k8s-cni-cncf-io\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885846    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-systemd\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885870    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnn92\" (UniqueName: \"kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92\") pod \"network-check-target-8q4hl\" (UID: \"1ebc8908-1ce3-489f-9401-49714efbaf34\") " pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885893    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-sys-fs\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885911    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-system-cni-dir\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885936    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-run-multus-certs\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.886856 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-tmp\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.887650 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885972    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8413e456-000f-457a-9ad6-a337819268cc-os-release\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.887650 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.885988    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af41025b-248c-4634-8c46-68d227fa3478-agent-certs\") pod \"konnectivity-agent-x5d7t\" (UID: \"af41025b-248c-4634-8c46-68d227fa3478\") " pod="kube-system/konnectivity-agent-x5d7t"
Jul 27 19:39:47.887650 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.886011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-sys\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.913950 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.913884    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 19:34:47 +0000 UTC" deadline="2028-03-22 15:13:24.54570333 +0000 UTC"
Jul 27 19:39:47.913950 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.913930    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14491h33m36.631776588s"
Jul 27 19:39:47.974643 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.974610    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 19:39:47.986437 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986401    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/003428df-67c2-4b3c-906a-260c8728b23a-kubelet-config\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:47.986631 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986445    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-kubernetes\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.986631 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986470    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-sysctl-d\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.986631 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986515    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-cni-bin\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.986631 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vtk2\" (UniqueName: \"kubernetes.io/projected/63bd477d-eac4-4ca6-b5d3-4734ba278992-kube-api-access-7vtk2\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.986631 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986564    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4btjr\" (UniqueName: \"kubernetes.io/projected/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-kube-api-access-4btjr\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.986631 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986571    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-kubernetes\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.986631 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986571    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/003428df-67c2-4b3c-906a-260c8728b23a-kubelet-config\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:47.986631 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986609    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-slash\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986648    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8413e456-000f-457a-9ad6-a337819268cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986674    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-etc-kubernetes\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-tuned\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986723    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/003428df-67c2-4b3c-906a-260c8728b23a-dbus\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986751    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00b00751-0c4b-431b-8699-6aad21590c67-tmp-dir\") pod \"node-resolver-bglxw\" (UID: \"00b00751-0c4b-431b-8699-6aad21590c67\") " pod="openshift-dns/node-resolver-bglxw"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986768    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-sysctl-d\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-var-lib-kubelet\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986803    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-run\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986828    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-cni-netd\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986856    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986897    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-modprobe-d\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986923    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-systemd-units\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.986980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987007    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9twl9\" (UniqueName: \"kubernetes.io/projected/ec6c229c-8276-42c2-9001-54332cc9da33-kube-api-access-9twl9\") pod \"iptables-alerter-xh555\" (UID: \"ec6c229c-8276-42c2-9001-54332cc9da33\") " pod="openshift-network-operator/iptables-alerter-xh555"
Jul 27 19:39:47.987050 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d50693b0-f760-4b28-8b31-514885f144ab-host\") pod \"node-ca-8qf2p\" (UID: \"d50693b0-f760-4b28-8b31-514885f144ab\") " pod="openshift-image-registry/node-ca-8qf2p"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-etc-selinux\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987084    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-multus-cni-dir\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-cnibin\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987134    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-os-release\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-sysconfig\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af41025b-248c-4634-8c46-68d227fa3478-konnectivity-ca\") pod \"konnectivity-agent-x5d7t\" (UID: \"af41025b-248c-4634-8c46-68d227fa3478\") " pod="kube-system/konnectivity-agent-x5d7t"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987225    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-registration-dir\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-device-dir\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987287    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-modprobe-d\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987354    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-var-lib-kubelet\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-etc-kubernetes\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-registration-dir\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987493    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-device-dir\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987528    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8413e456-000f-457a-9ad6-a337819268cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.987866 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987636    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-sysconfig\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.988628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.987276    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-var-lib-kubelet\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.988628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.988006    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af41025b-248c-4634-8c46-68d227fa3478-konnectivity-ca\") pod \"konnectivity-agent-x5d7t\" (UID: \"af41025b-248c-4634-8c46-68d227fa3478\") " pod="kube-system/konnectivity-agent-x5d7t"
Jul 27 19:39:47.988628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.988052    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 19:39:47.988628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.988085    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-var-lib-kubelet\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.988628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.988098    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-os-release\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.988628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.988137    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d50693b0-f760-4b28-8b31-514885f144ab-host\") pod \"node-ca-8qf2p\" (UID: \"d50693b0-f760-4b28-8b31-514885f144ab\") " pod="openshift-image-registry/node-ca-8qf2p"
Jul 27 19:39:47.988628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.988194    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-run\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.988628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.988242    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-multus-cni-dir\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.988628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.988247    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/003428df-67c2-4b3c-906a-260c8728b23a-dbus\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:47.988628 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:47.988275    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:39:47.988628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.988326    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-etc-selinux\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.988628 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:47.988356    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret podName:003428df-67c2-4b3c-906a-260c8728b23a nodeName:}" failed. No retries permitted until 2026-07-27 19:39:48.488334659 +0000 UTC m=+3.040251080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret") pod "global-pull-secret-syncer-jbl6q" (UID: "003428df-67c2-4b3c-906a-260c8728b23a") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:39:47.988628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.988366    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-cnibin\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.988628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.988418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3dcb99e5-a045-4433-a3dc-2851abcba37a-env-overrides\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.989156 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.988995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3dcb99e5-a045-4433-a3dc-2851abcba37a-ovnkube-script-lib\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.989156 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs\") pod \"network-metrics-daemon-z2n28\" (UID: \"cb8b05dc-fed1-4ced-b993-489c5da47ecf\") " pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:47.989156 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989113    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8413e456-000f-457a-9ad6-a337819268cc-system-cni-dir\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.989156 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989140    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshb5\" (UniqueName: \"kubernetes.io/projected/00b00751-0c4b-431b-8699-6aad21590c67-kube-api-access-lshb5\") pod \"node-resolver-bglxw\" (UID: \"00b00751-0c4b-431b-8699-6aad21590c67\") " pod="openshift-dns/node-resolver-bglxw"
Jul 27 19:39:47.989302 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-var-lib-cni-multus\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.989302 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989205    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-etc-openvswitch\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.989302 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6hqg\" (UniqueName: \"kubernetes.io/projected/3dcb99e5-a045-4433-a3dc-2851abcba37a-kube-api-access-t6hqg\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.989302 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989288    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8413e456-000f-457a-9ad6-a337819268cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.989446 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989316    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-multus-socket-dir-parent\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.989446 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989350    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-run-k8s-cni-cncf-io\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.989446 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989383    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-systemd\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.989446 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-var-lib-openvswitch\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.989606 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989452    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-run-openvswitch\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.989606 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989481    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnn92\" (UniqueName: \"kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92\") pod \"network-check-target-8q4hl\" (UID: \"1ebc8908-1ce3-489f-9401-49714efbaf34\") " pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:39:47.989606 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989514    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-sys-fs\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.989606 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.989545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-system-cni-dir\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.989606 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.988504    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00b00751-0c4b-431b-8699-6aad21590c67-tmp-dir\") pod \"node-resolver-bglxw\" (UID: \"00b00751-0c4b-431b-8699-6aad21590c67\") " pod="openshift-dns/node-resolver-bglxw"
Jul 27 19:39:47.990443 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.990416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-run-multus-certs\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.990646 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.990621    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-run-k8s-cni-cncf-io\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.990746 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.990654    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-run-multus-certs\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.990746 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.990730    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-system-cni-dir\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.990859 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.990788    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-tmp\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.990859 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.990821    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-run-ovn\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.990960 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.990860    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-log-socket\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.990960 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.990899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8413e456-000f-457a-9ad6-a337819268cc-os-release\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.990960 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.990904    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8413e456-000f-457a-9ad6-a337819268cc-system-cni-dir\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.991117 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991003    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af41025b-248c-4634-8c46-68d227fa3478-agent-certs\") pod \"konnectivity-agent-x5d7t\" (UID: \"af41025b-248c-4634-8c46-68d227fa3478\") " pod="kube-system/konnectivity-agent-x5d7t"
Jul 27 19:39:47.991117 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991043    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-sys\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.991117 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991083    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec6c229c-8276-42c2-9001-54332cc9da33-host-slash\") pod \"iptables-alerter-xh555\" (UID: \"ec6c229c-8276-42c2-9001-54332cc9da33\") " pod="openshift-network-operator/iptables-alerter-xh555"
Jul 27 19:39:47.991270 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.991270 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991135    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-systemd\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.991270 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991152    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-socket-dir\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.991414 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991395    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-var-lib-cni-bin\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.991471 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991428    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-run-netns\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.991539 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991466    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3dcb99e5-a045-4433-a3dc-2851abcba37a-ovn-node-metrics-cert\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.991539 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991510    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-socket-dir\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.991539 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991524    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec6c229c-8276-42c2-9001-54332cc9da33-iptables-alerter-script\") pod \"iptables-alerter-xh555\" (UID: \"ec6c229c-8276-42c2-9001-54332cc9da33\") " pod="openshift-network-operator/iptables-alerter-xh555"
Jul 27 19:39:47.991770 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d50693b0-f760-4b28-8b31-514885f144ab-serviceca\") pod \"node-ca-8qf2p\" (UID: \"d50693b0-f760-4b28-8b31-514885f144ab\") " pod="openshift-image-registry/node-ca-8qf2p"
Jul 27 19:39:47.991770 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991615    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjb7x\" (UniqueName: \"kubernetes.io/projected/d50693b0-f760-4b28-8b31-514885f144ab-kube-api-access-sjb7x\") pod \"node-ca-8qf2p\" (UID: \"d50693b0-f760-4b28-8b31-514885f144ab\") " pod="openshift-image-registry/node-ca-8qf2p"
Jul 27 19:39:47.991770 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:47.991630    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:39:47.991770 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991652    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwlcb\" (UniqueName: \"kubernetes.io/projected/2d58052c-f824-4f53-8459-6c6f16e4bac9-kube-api-access-dwlcb\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.991770 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991688    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00b00751-0c4b-431b-8699-6aad21590c67-hosts-file\") pod \"node-resolver-bglxw\" (UID: \"00b00751-0c4b-431b-8699-6aad21590c67\") " pod="openshift-dns/node-resolver-bglxw"
Jul 27 19:39:47.991770 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:47.991702    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs podName:cb8b05dc-fed1-4ced-b993-489c5da47ecf nodeName:}" failed. No retries permitted until 2026-07-27 19:39:48.491684024 +0000 UTC m=+3.043600437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs") pod "network-metrics-daemon-z2n28" (UID: "cb8b05dc-fed1-4ced-b993-489c5da47ecf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:39:47.991770 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.991740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-kubelet\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-run-systemd\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8413e456-000f-457a-9ad6-a337819268cc-cnibin\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992261    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8413e456-000f-457a-9ad6-a337819268cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-run-netns\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992301    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-hostroot\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992327    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-lib-modules\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992350    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-host\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8413e456-000f-457a-9ad6-a337819268cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992402    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-multus-conf-dir\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992435    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63bd477d-eac4-4ca6-b5d3-4734ba278992-multus-daemon-config\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992458    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-node-log\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992482    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc74q\" (UniqueName: \"kubernetes.io/projected/cb8b05dc-fed1-4ced-b993-489c5da47ecf-kube-api-access-fc74q\") pod \"network-metrics-daemon-z2n28\" (UID: \"cb8b05dc-fed1-4ced-b993-489c5da47ecf\") " pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992520    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcxg\" (UniqueName: \"kubernetes.io/projected/8413e456-000f-457a-9ad6-a337819268cc-kube-api-access-wvcxg\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63bd477d-eac4-4ca6-b5d3-4734ba278992-cni-binary-copy\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-sysctl-conf\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992605    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3dcb99e5-a045-4433-a3dc-2851abcba37a-ovnkube-config\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:47.995417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992715    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00b00751-0c4b-431b-8699-6aad21590c67-hosts-file\") pod \"node-resolver-bglxw\" (UID: \"00b00751-0c4b-431b-8699-6aad21590c67\") " pod="openshift-dns/node-resolver-bglxw"
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.992759    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.993202    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8413e456-000f-457a-9ad6-a337819268cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.993260    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-var-lib-cni-multus\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.993328    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8413e456-000f-457a-9ad6-a337819268cc-os-release\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.993364    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-multus-socket-dir-parent\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.993400    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-var-lib-cni-bin\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.993762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d50693b0-f760-4b28-8b31-514885f144ab-serviceca\") pod \"node-ca-8qf2p\" (UID: \"d50693b0-f760-4b28-8b31-514885f144ab\") " pod="openshift-image-registry/node-ca-8qf2p"
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.994426    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-tuned\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.994541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-lib-modules\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.994598    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2d58052c-f824-4f53-8459-6c6f16e4bac9-sys-fs\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:47.995946    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:47.995968    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:47.995990    2562 projected.go:196] Error preparing data for projected volume kube-api-access-pnn92 for pod openshift-network-diagnostics/network-check-target-8q4hl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.996092    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-host\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.996331 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.996165    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-multus-conf-dir\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.997146 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.996423    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8413e456-000f-457a-9ad6-a337819268cc-cnibin\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.997146 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.996503    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-sys\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.997146 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.996711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8413e456-000f-457a-9ad6-a337819268cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.997146 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.996734    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8413e456-000f-457a-9ad6-a337819268cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:47.997146 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.996779    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-host-run-netns\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.997146 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.997090    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63bd477d-eac4-4ca6-b5d3-4734ba278992-hostroot\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.997449 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.997209    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63bd477d-eac4-4ca6-b5d3-4734ba278992-multus-daemon-config\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:47.997449 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:47.997276    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92 podName:1ebc8908-1ce3-489f-9401-49714efbaf34 nodeName:}" failed. No retries permitted until 2026-07-27 19:39:48.497257245 +0000 UTC m=+3.049173642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pnn92" (UniqueName: "kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92") pod "network-check-target-8q4hl" (UID: "1ebc8908-1ce3-489f-9401-49714efbaf34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:39:47.997449 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.997214    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-etc-sysctl-conf\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:47.997449 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:47.997340    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63bd477d-eac4-4ca6-b5d3-4734ba278992-cni-binary-copy\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:48.001725 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.001697    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshb5\" (UniqueName: \"kubernetes.io/projected/00b00751-0c4b-431b-8699-6aad21590c67-kube-api-access-lshb5\") pod \"node-resolver-bglxw\" (UID: \"00b00751-0c4b-431b-8699-6aad21590c67\") " pod="openshift-dns/node-resolver-bglxw"
Jul 27 19:39:48.001725 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.001716    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-tmp\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:48.002164 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.002142    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vtk2\" (UniqueName: \"kubernetes.io/projected/63bd477d-eac4-4ca6-b5d3-4734ba278992-kube-api-access-7vtk2\") pod \"multus-42l2f\" (UID: \"63bd477d-eac4-4ca6-b5d3-4734ba278992\") " pod="openshift-multus/multus-42l2f"
Jul 27 19:39:48.002493 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.002465    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af41025b-248c-4634-8c46-68d227fa3478-agent-certs\") pod \"konnectivity-agent-x5d7t\" (UID: \"af41025b-248c-4634-8c46-68d227fa3478\") " pod="kube-system/konnectivity-agent-x5d7t"
Jul 27 19:39:48.004013 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.003991    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4btjr\" (UniqueName: \"kubernetes.io/projected/e3bc4cbd-3f86-4404-a4f2-10c765d2bd32-kube-api-access-4btjr\") pod \"tuned-gm5lr\" (UID: \"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32\") " pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:48.007152 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.007128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwlcb\" (UniqueName: \"kubernetes.io/projected/2d58052c-f824-4f53-8459-6c6f16e4bac9-kube-api-access-dwlcb\") pod \"aws-ebs-csi-driver-node-bnrrf\" (UID: \"2d58052c-f824-4f53-8459-6c6f16e4bac9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:48.007265 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.007131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc74q\" (UniqueName: \"kubernetes.io/projected/cb8b05dc-fed1-4ced-b993-489c5da47ecf-kube-api-access-fc74q\") pod \"network-metrics-daemon-z2n28\" (UID: \"cb8b05dc-fed1-4ced-b993-489c5da47ecf\") " pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:48.007488 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.007447    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvcxg\" (UniqueName: \"kubernetes.io/projected/8413e456-000f-457a-9ad6-a337819268cc-kube-api-access-wvcxg\") pod \"multus-additional-cni-plugins-7mwwb\" (UID: \"8413e456-000f-457a-9ad6-a337819268cc\") " pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:48.009998 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.009974    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjb7x\" (UniqueName: \"kubernetes.io/projected/d50693b0-f760-4b28-8b31-514885f144ab-kube-api-access-sjb7x\") pod \"node-ca-8qf2p\" (UID: \"d50693b0-f760-4b28-8b31-514885f144ab\") " pod="openshift-image-registry/node-ca-8qf2p"
Jul 27 19:39:48.093057 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.092958    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-var-lib-openvswitch\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.093057 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093011    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-run-openvswitch\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.093287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093063    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-run-ovn\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.093287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093074    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-var-lib-openvswitch\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.093287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093083    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-log-socket\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.093287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093130    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-log-socket\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.093287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093142    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec6c229c-8276-42c2-9001-54332cc9da33-host-slash\") pod \"iptables-alerter-xh555\" (UID: \"ec6c229c-8276-42c2-9001-54332cc9da33\") " pod="openshift-network-operator/iptables-alerter-xh555"
Jul 27 19:39:48.093287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093173    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-run-netns\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.093287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093180    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-run-openvswitch\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.093287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3dcb99e5-a045-4433-a3dc-2851abcba37a-ovn-node-metrics-cert\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.093287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec6c229c-8276-42c2-9001-54332cc9da33-iptables-alerter-script\") pod \"iptables-alerter-xh555\" (UID: \"ec6c229c-8276-42c2-9001-54332cc9da33\") " pod="openshift-network-operator/iptables-alerter-xh555"
Jul 27 19:39:48.093287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093217    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-run-ovn\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.093287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-kubelet\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.093287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093258    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-run-systemd\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.093287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093268    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-run-netns\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.093287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-node-log\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093291    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec6c229c-8276-42c2-9001-54332cc9da33-host-slash\") pod \"iptables-alerter-xh555\" (UID: \"ec6c229c-8276-42c2-9001-54332cc9da33\") " pod="openshift-network-operator/iptables-alerter-xh555"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3dcb99e5-a045-4433-a3dc-2851abcba37a-ovnkube-config\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093355    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-cni-bin\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-slash\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093417    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-cni-netd\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093452    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-systemd-units\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093478    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093556    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9twl9\" (UniqueName: \"kubernetes.io/projected/ec6c229c-8276-42c2-9001-54332cc9da33-kube-api-access-9twl9\") pod \"iptables-alerter-xh555\" (UID: \"ec6c229c-8276-42c2-9001-54332cc9da33\") " pod="openshift-network-operator/iptables-alerter-xh555"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093601    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3dcb99e5-a045-4433-a3dc-2851abcba37a-env-overrides\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3dcb99e5-a045-4433-a3dc-2851abcba37a-ovnkube-script-lib\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093651    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-etc-openvswitch\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6hqg\" (UniqueName: \"kubernetes.io/projected/3dcb99e5-a045-4433-a3dc-2851abcba37a-kube-api-access-t6hqg\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093920    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3dcb99e5-a045-4433-a3dc-2851abcba37a-ovnkube-config\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093924    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec6c229c-8276-42c2-9001-54332cc9da33-iptables-alerter-script\") pod \"iptables-alerter-xh555\" (UID: \"ec6c229c-8276-42c2-9001-54332cc9da33\") " pod="openshift-network-operator/iptables-alerter-xh555"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093951    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-cni-bin\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.094032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093987    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-run-systemd\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.095119 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.093997    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-kubelet\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.095119 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.094019    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-node-log\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.095119 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.094039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-slash\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.095119 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.094059    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.095119 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.094108    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-cni-netd\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.095119 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.094148    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-systemd-units\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.095119 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.094181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.095119 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.094285    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dcb99e5-a045-4433-a3dc-2851abcba37a-etc-openvswitch\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.095119 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.094695    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3dcb99e5-a045-4433-a3dc-2851abcba37a-ovnkube-script-lib\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.095119 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.094709    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3dcb99e5-a045-4433-a3dc-2851abcba37a-env-overrides\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.096783 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.096756    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3dcb99e5-a045-4433-a3dc-2851abcba37a-ovn-node-metrics-cert\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.103678 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.103647    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6hqg\" (UniqueName: \"kubernetes.io/projected/3dcb99e5-a045-4433-a3dc-2851abcba37a-kube-api-access-t6hqg\") pod \"ovnkube-node-8bnpc\" (UID: \"3dcb99e5-a045-4433-a3dc-2851abcba37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.103803 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.103707    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9twl9\" (UniqueName: \"kubernetes.io/projected/ec6c229c-8276-42c2-9001-54332cc9da33-kube-api-access-9twl9\") pod \"iptables-alerter-xh555\" (UID: \"ec6c229c-8276-42c2-9001-54332cc9da33\") " pod="openshift-network-operator/iptables-alerter-xh555"
Jul 27 19:39:48.150139 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.150098    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:39:48.163914 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.163880    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8qf2p"
Jul 27 19:39:48.175483 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.175458    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-42l2f"
Jul 27 19:39:48.188616 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.188594    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7mwwb"
Jul 27 19:39:48.193638 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.193617    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x5d7t"
Jul 27 19:39:48.201641 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.201620    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf"
Jul 27 19:39:48.210632 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.210609    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gm5lr"
Jul 27 19:39:48.219531 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.219504    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bglxw"
Jul 27 19:39:48.226500 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.226461    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:39:48.235431 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.235410    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xh555"
Jul 27 19:39:48.498370 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.498276    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs\") pod \"network-metrics-daemon-z2n28\" (UID: \"cb8b05dc-fed1-4ced-b993-489c5da47ecf\") " pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:48.498370 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.498328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnn92\" (UniqueName: \"kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92\") pod \"network-check-target-8q4hl\" (UID: \"1ebc8908-1ce3-489f-9401-49714efbaf34\") " pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:39:48.498644 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.498388    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:48.498644 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:48.498464    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:39:48.498644 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:48.498533    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:39:48.498644 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:48.498539    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:39:48.498644 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:48.498564    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:39:48.498644 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:48.498579    2562 projected.go:196] Error preparing data for projected volume kube-api-access-pnn92 for pod openshift-network-diagnostics/network-check-target-8q4hl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:39:48.498644 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:48.498564    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs podName:cb8b05dc-fed1-4ced-b993-489c5da47ecf nodeName:}" failed. No retries permitted until 2026-07-27 19:39:49.498542874 +0000 UTC m=+4.050459275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs") pod "network-metrics-daemon-z2n28" (UID: "cb8b05dc-fed1-4ced-b993-489c5da47ecf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:39:48.498644 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:48.498637    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret podName:003428df-67c2-4b3c-906a-260c8728b23a nodeName:}" failed. No retries permitted until 2026-07-27 19:39:49.498621819 +0000 UTC m=+4.050538220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret") pod "global-pull-secret-syncer-jbl6q" (UID: "003428df-67c2-4b3c-906a-260c8728b23a") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:39:48.498644 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:48.498650    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92 podName:1ebc8908-1ce3-489f-9401-49714efbaf34 nodeName:}" failed. No retries permitted until 2026-07-27 19:39:49.498643985 +0000 UTC m=+4.050560383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnn92" (UniqueName: "kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92") pod "network-check-target-8q4hl" (UID: "1ebc8908-1ce3-489f-9401-49714efbaf34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:39:48.914366 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.914300    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 19:34:47 +0000 UTC" deadline="2028-01-07 20:22:05.032134045 +0000 UTC"
Jul 27 19:39:48.914366 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:48.914363    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12696h42m16.117775905s"
Jul 27 19:39:49.019973 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:49.019946    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6c229c_8276_42c2_9001_54332cc9da33.slice/crio-6c238af7fa1827805cd27ec863cbc15905afffc0687aa2907c25c7b3ea1dc028 WatchSource:0}: Error finding container 6c238af7fa1827805cd27ec863cbc15905afffc0687aa2907c25c7b3ea1dc028: Status 404 returned error can't find the container with id 6c238af7fa1827805cd27ec863cbc15905afffc0687aa2907c25c7b3ea1dc028
Jul 27 19:39:49.056705 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:49.056674    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b00751_0c4b_431b_8699_6aad21590c67.slice/crio-a7e3c857c7fb5a7e921d1a92828663185654bd2b35d1eff79e0d1ab314879974 WatchSource:0}: Error finding container a7e3c857c7fb5a7e921d1a92828663185654bd2b35d1eff79e0d1ab314879974: Status 404 returned error can't find the container with id a7e3c857c7fb5a7e921d1a92828663185654bd2b35d1eff79e0d1ab314879974
Jul 27 19:39:49.057006 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:49.056985    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8413e456_000f_457a_9ad6_a337819268cc.slice/crio-a8fe8179407ee834d83ec232bc29a2f3b298eca7a56693115be918753eb4c7a8 WatchSource:0}: Error finding container a8fe8179407ee834d83ec232bc29a2f3b298eca7a56693115be918753eb4c7a8: Status 404 returned error can't find the container with id a8fe8179407ee834d83ec232bc29a2f3b298eca7a56693115be918753eb4c7a8
Jul 27 19:39:49.057873 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:49.057848    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d58052c_f824_4f53_8459_6c6f16e4bac9.slice/crio-0ea39625e6e35beee67486d0a804eabec48a77ede6a8ba3f589335b9d1484222 WatchSource:0}: Error finding container 0ea39625e6e35beee67486d0a804eabec48a77ede6a8ba3f589335b9d1484222: Status 404 returned error can't find the container with id 0ea39625e6e35beee67486d0a804eabec48a77ede6a8ba3f589335b9d1484222
Jul 27 19:39:49.058671 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:49.058588    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bc4cbd_3f86_4404_a4f2_10c765d2bd32.slice/crio-8a06c4b832364e77618226c5b5354322b637b5f47782654d856f25b11a20d888 WatchSource:0}: Error finding container 8a06c4b832364e77618226c5b5354322b637b5f47782654d856f25b11a20d888: Status 404 returned error can't find the container with id 8a06c4b832364e77618226c5b5354322b637b5f47782654d856f25b11a20d888
Jul 27 19:39:49.061150 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:49.061123    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf41025b_248c_4634_8c46_68d227fa3478.slice/crio-2b474cf5113e5851ed3db547d42d58a5818b29d5e2d4526390226a3ed05ad88e WatchSource:0}: Error finding container 2b474cf5113e5851ed3db547d42d58a5818b29d5e2d4526390226a3ed05ad88e: Status 404 returned error can't find the container with id 2b474cf5113e5851ed3db547d42d58a5818b29d5e2d4526390226a3ed05ad88e
Jul 27 19:39:49.062484 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:49.062459    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50693b0_f760_4b28_8b31_514885f144ab.slice/crio-a1109997004db47118be22757f6d7a1436d6f100d7eaba815ed47da646959292 WatchSource:0}: Error finding container a1109997004db47118be22757f6d7a1436d6f100d7eaba815ed47da646959292: Status 404 returned error can't find the container with id a1109997004db47118be22757f6d7a1436d6f100d7eaba815ed47da646959292
Jul 27 19:39:49.062750 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:49.062726    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bd477d_eac4_4ca6_b5d3_4734ba278992.slice/crio-6bfab725bbe7383b22debcc067fdce1ba814262c6d3023de7c505f9c15376b77 WatchSource:0}: Error finding container 6bfab725bbe7383b22debcc067fdce1ba814262c6d3023de7c505f9c15376b77: Status 404 returned error can't find the container with id 6bfab725bbe7383b22debcc067fdce1ba814262c6d3023de7c505f9c15376b77
Jul 27 19:39:49.064365 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:39:49.063950    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcb99e5_a045_4433_a3dc_2851abcba37a.slice/crio-4eb1f3282f7d1049d9d5c949970a7a583787fe2613910da480f0bb254f0c91a6 WatchSource:0}: Error finding container 4eb1f3282f7d1049d9d5c949970a7a583787fe2613910da480f0bb254f0c91a6: Status 404 returned error can't find the container with id 4eb1f3282f7d1049d9d5c949970a7a583787fe2613910da480f0bb254f0c91a6
Jul 27 19:39:49.512147 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:49.512106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:49.512319 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:49.512172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs\") pod \"network-metrics-daemon-z2n28\" (UID: \"cb8b05dc-fed1-4ced-b993-489c5da47ecf\") " pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:49.512319 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:49.512206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnn92\" (UniqueName: \"kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92\") pod \"network-check-target-8q4hl\" (UID: \"1ebc8908-1ce3-489f-9401-49714efbaf34\") " pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:39:49.512319 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:49.512258    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:39:49.512434 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:49.512330    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret podName:003428df-67c2-4b3c-906a-260c8728b23a nodeName:}" failed. No retries permitted until 2026-07-27 19:39:51.51231059 +0000 UTC m=+6.064226990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret") pod "global-pull-secret-syncer-jbl6q" (UID: "003428df-67c2-4b3c-906a-260c8728b23a") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:39:49.512434 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:49.512355    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:39:49.512434 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:49.512370    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:39:49.512434 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:49.512383    2562 projected.go:196] Error preparing data for projected volume kube-api-access-pnn92 for pod openshift-network-diagnostics/network-check-target-8q4hl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:39:49.512434 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:49.512421    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92 podName:1ebc8908-1ce3-489f-9401-49714efbaf34 nodeName:}" failed. No retries permitted until 2026-07-27 19:39:51.512408672 +0000 UTC m=+6.064325077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnn92" (UniqueName: "kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92") pod "network-check-target-8q4hl" (UID: "1ebc8908-1ce3-489f-9401-49714efbaf34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:39:49.512670 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:49.512487    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:39:49.512670 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:49.512542    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs podName:cb8b05dc-fed1-4ced-b993-489c5da47ecf nodeName:}" failed. No retries permitted until 2026-07-27 19:39:51.512530909 +0000 UTC m=+6.064447309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs") pod "network-metrics-daemon-z2n28" (UID: "cb8b05dc-fed1-4ced-b993-489c5da47ecf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:39:49.985792 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:49.985710    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:49.986264 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:49.985865    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:39:49.986339 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:49.986314    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:49.986435 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:49.986409    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:39:49.988959 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:49.988763    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:39:49.988959 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:49.988868    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:39:50.018391 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:50.018357    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xh555" event={"ID":"ec6c229c-8276-42c2-9001-54332cc9da33","Type":"ContainerStarted","Data":"6c238af7fa1827805cd27ec863cbc15905afffc0687aa2907c25c7b3ea1dc028"}
Jul 27 19:39:50.030670 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:50.030636    2562 generic.go:356] "Generic (PLEG): container finished" podID="580850b6194cd2dae03e2367ddc94987" containerID="98846c8aacaf9837ddff2f5dd8cc89bbb842fa13bd9ca6d1b8c63a4007859568" exitCode=0
Jul 27 19:39:50.030874 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:50.030762    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" event={"ID":"580850b6194cd2dae03e2367ddc94987","Type":"ContainerDied","Data":"98846c8aacaf9837ddff2f5dd8cc89bbb842fa13bd9ca6d1b8c63a4007859568"}
Jul 27 19:39:50.039832 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:50.039782    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc" event={"ID":"3dcb99e5-a045-4433-a3dc-2851abcba37a","Type":"ContainerStarted","Data":"4eb1f3282f7d1049d9d5c949970a7a583787fe2613910da480f0bb254f0c91a6"}
Jul 27 19:39:50.043666 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:50.043626    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-42l2f" event={"ID":"63bd477d-eac4-4ca6-b5d3-4734ba278992","Type":"ContainerStarted","Data":"6bfab725bbe7383b22debcc067fdce1ba814262c6d3023de7c505f9c15376b77"}
Jul 27 19:39:50.048280 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:50.048227    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x5d7t" event={"ID":"af41025b-248c-4634-8c46-68d227fa3478","Type":"ContainerStarted","Data":"2b474cf5113e5851ed3db547d42d58a5818b29d5e2d4526390226a3ed05ad88e"}
Jul 27 19:39:50.061882 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:50.061849    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gm5lr" event={"ID":"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32","Type":"ContainerStarted","Data":"8a06c4b832364e77618226c5b5354322b637b5f47782654d856f25b11a20d888"}
Jul 27 19:39:50.069069 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:50.069040    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf" event={"ID":"2d58052c-f824-4f53-8459-6c6f16e4bac9","Type":"ContainerStarted","Data":"0ea39625e6e35beee67486d0a804eabec48a77ede6a8ba3f589335b9d1484222"}
Jul 27 19:39:50.072639 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:50.072590    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mwwb" event={"ID":"8413e456-000f-457a-9ad6-a337819268cc","Type":"ContainerStarted","Data":"a8fe8179407ee834d83ec232bc29a2f3b298eca7a56693115be918753eb4c7a8"}
Jul 27 19:39:50.081540 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:50.081506    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal" event={"ID":"028ba8eadc770529daa4c0f4fe338d90","Type":"ContainerStarted","Data":"2924d848be04dd6e885f064dd88cf98d12ab3502abbc80d016016407b8ff9624"}
Jul 27 19:39:50.084281 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:50.084252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8qf2p" event={"ID":"d50693b0-f760-4b28-8b31-514885f144ab","Type":"ContainerStarted","Data":"a1109997004db47118be22757f6d7a1436d6f100d7eaba815ed47da646959292"}
Jul 27 19:39:50.087520 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:50.087460    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bglxw" event={"ID":"00b00751-0c4b-431b-8699-6aad21590c67","Type":"ContainerStarted","Data":"a7e3c857c7fb5a7e921d1a92828663185654bd2b35d1eff79e0d1ab314879974"}
Jul 27 19:39:51.098014 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:51.097964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" event={"ID":"580850b6194cd2dae03e2367ddc94987","Type":"ContainerStarted","Data":"356aea74226daa609371c22f680d819d61fbbba24728740a47af0af3a4542dec"}
Jul 27 19:39:51.117282 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:51.117165    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-140.ec2.internal" podStartSLOduration=4.117143815 podStartE2EDuration="4.117143815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:39:50.09945395 +0000 UTC m=+4.651370374" watchObservedRunningTime="2026-07-27 19:39:51.117143815 +0000 UTC m=+5.669060238"
Jul 27 19:39:51.532882 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:51.532784    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs\") pod \"network-metrics-daemon-z2n28\" (UID: \"cb8b05dc-fed1-4ced-b993-489c5da47ecf\") " pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:51.532882 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:51.532855    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnn92\" (UniqueName: \"kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92\") pod \"network-check-target-8q4hl\" (UID: \"1ebc8908-1ce3-489f-9401-49714efbaf34\") " pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:39:51.533114 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:51.532916    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:51.533114 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:51.533048    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:39:51.533232 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:51.533116    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret podName:003428df-67c2-4b3c-906a-260c8728b23a nodeName:}" failed. No retries permitted until 2026-07-27 19:39:55.533095908 +0000 UTC m=+10.085012310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret") pod "global-pull-secret-syncer-jbl6q" (UID: "003428df-67c2-4b3c-906a-260c8728b23a") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:39:51.533793 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:51.533532    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:39:51.533793 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:51.533606    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs podName:cb8b05dc-fed1-4ced-b993-489c5da47ecf nodeName:}" failed. No retries permitted until 2026-07-27 19:39:55.533588869 +0000 UTC m=+10.085505272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs") pod "network-metrics-daemon-z2n28" (UID: "cb8b05dc-fed1-4ced-b993-489c5da47ecf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:39:51.533793 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:51.533689    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:39:51.533793 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:51.533707    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:39:51.533793 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:51.533720    2562 projected.go:196] Error preparing data for projected volume kube-api-access-pnn92 for pod openshift-network-diagnostics/network-check-target-8q4hl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:39:51.533793 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:51.533755    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92 podName:1ebc8908-1ce3-489f-9401-49714efbaf34 nodeName:}" failed. No retries permitted until 2026-07-27 19:39:55.533742975 +0000 UTC m=+10.085659380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnn92" (UniqueName: "kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92") pod "network-check-target-8q4hl" (UID: "1ebc8908-1ce3-489f-9401-49714efbaf34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:39:51.988348 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:51.987444    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:51.988348 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:51.987599    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:39:51.988348 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:51.987998    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:39:51.988348 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:51.988096    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:39:51.988348 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:51.988151    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:51.988348 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:51.988236    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:39:53.987483 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:53.986692    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:53.987483 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:53.986848    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:39:53.987483 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:53.987337    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:53.987483 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:53.987436    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:39:53.988374 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:53.988240    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:39:53.988374 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:53.988337    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:39:55.567444 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:55.567406    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:55.568027 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:55.567464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs\") pod \"network-metrics-daemon-z2n28\" (UID: \"cb8b05dc-fed1-4ced-b993-489c5da47ecf\") " pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:55.568027 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:55.567499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnn92\" (UniqueName: \"kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92\") pod \"network-check-target-8q4hl\" (UID: \"1ebc8908-1ce3-489f-9401-49714efbaf34\") " pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:39:55.568027 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:55.567670    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:39:55.568027 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:55.567689    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:39:55.568027 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:55.567703    2562 projected.go:196] Error preparing data for projected volume kube-api-access-pnn92 for pod openshift-network-diagnostics/network-check-target-8q4hl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:39:55.568027 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:55.567764    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92 podName:1ebc8908-1ce3-489f-9401-49714efbaf34 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:03.567744844 +0000 UTC m=+18.119661264 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnn92" (UniqueName: "kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92") pod "network-check-target-8q4hl" (UID: "1ebc8908-1ce3-489f-9401-49714efbaf34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:39:55.568279 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:55.568169    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:39:55.568279 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:55.568215    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret podName:003428df-67c2-4b3c-906a-260c8728b23a nodeName:}" failed. No retries permitted until 2026-07-27 19:40:03.56820042 +0000 UTC m=+18.120116838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret") pod "global-pull-secret-syncer-jbl6q" (UID: "003428df-67c2-4b3c-906a-260c8728b23a") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:39:55.568279 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:55.568258    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:39:55.568384 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:55.568281    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs podName:cb8b05dc-fed1-4ced-b993-489c5da47ecf nodeName:}" failed. No retries permitted until 2026-07-27 19:40:03.568273672 +0000 UTC m=+18.120190070 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs") pod "network-metrics-daemon-z2n28" (UID: "cb8b05dc-fed1-4ced-b993-489c5da47ecf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:39:55.986703 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:55.986608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:39:55.986703 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:55.986612    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:55.987883 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:55.987502    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:39:55.987883 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:55.987643    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:55.987883 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:55.987632    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:39:55.987883 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:55.987729    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:39:57.987687 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:57.987645    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:57.988183 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:57.987798    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:39:57.988275 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:57.988236    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:57.988356 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:57.988333    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:39:57.988416 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:57.988402    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:39:57.988478 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:57.988455    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:39:59.985523 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:59.985472    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:39:59.986116 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:59.985629    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:39:59.986116 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:59.985992    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:39:59.986116 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:59.986066    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:39:59.986328 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:39:59.986177    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:39:59.986328 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:39:59.986241    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:40:01.984891 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:01.984856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:01.985343 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:01.984856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:01.985343 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:01.984975    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:40:01.985343 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:01.984856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:01.985343 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:01.985072    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:40:01.985343 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:01.985134    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:40:03.630799 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:03.630760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:03.630799 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:03.630815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs\") pod \"network-metrics-daemon-z2n28\" (UID: \"cb8b05dc-fed1-4ced-b993-489c5da47ecf\") " pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:03.631424 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:03.630852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnn92\" (UniqueName: \"kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92\") pod \"network-check-target-8q4hl\" (UID: \"1ebc8908-1ce3-489f-9401-49714efbaf34\") " pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:03.631424 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:03.630935    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:40:03.631424 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:03.630985    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:40:03.631424 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:03.631006    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:40:03.631424 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:03.631013    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret podName:003428df-67c2-4b3c-906a-260c8728b23a nodeName:}" failed. No retries permitted until 2026-07-27 19:40:19.630988835 +0000 UTC m=+34.182905234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret") pod "global-pull-secret-syncer-jbl6q" (UID: "003428df-67c2-4b3c-906a-260c8728b23a") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:40:03.631424 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:03.631017    2562 projected.go:196] Error preparing data for projected volume kube-api-access-pnn92 for pod openshift-network-diagnostics/network-check-target-8q4hl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:40:03.631424 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:03.631056    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92 podName:1ebc8908-1ce3-489f-9401-49714efbaf34 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:19.631046241 +0000 UTC m=+34.182962637 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnn92" (UniqueName: "kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92") pod "network-check-target-8q4hl" (UID: "1ebc8908-1ce3-489f-9401-49714efbaf34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:40:03.631424 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:03.631061    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:40:03.631424 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:03.631100    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs podName:cb8b05dc-fed1-4ced-b993-489c5da47ecf nodeName:}" failed. No retries permitted until 2026-07-27 19:40:19.631085671 +0000 UTC m=+34.183002066 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs") pod "network-metrics-daemon-z2n28" (UID: "cb8b05dc-fed1-4ced-b993-489c5da47ecf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:40:03.985149 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:03.985073    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:03.985286 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:03.985186    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:40:03.985286 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:03.985073    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:03.985286 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:03.985259    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:40:03.985286 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:03.985073    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:03.985445 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:03.985302    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:40:05.986012 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:05.985530    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:05.986012 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:05.985561    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:05.986012 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:05.985655    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:40:05.986012 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:05.985690    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:05.986012 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:05.985784    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:40:05.986012 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:05.985857    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:40:07.985104 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:07.985074    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:07.985104 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:07.985096    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:07.985685 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:07.985085    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:07.985685 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:07.985194    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:40:07.985685 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:07.985280    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:40:07.985685 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:07.985355    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:40:09.140017 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.139637    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc" event={"ID":"3dcb99e5-a045-4433-a3dc-2851abcba37a","Type":"ContainerStarted","Data":"ba9da0b246f20ed4d105ebd818432e44626d487ca7f9c33622ab584427dac67a"}
Jul 27 19:40:09.140847 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.140118    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc" event={"ID":"3dcb99e5-a045-4433-a3dc-2851abcba37a","Type":"ContainerStarted","Data":"9b58595e683ed9ce023d135b80f6485b43ba5faf60f08458a5c3d735fa091c5d"}
Jul 27 19:40:09.140847 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.140166    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc" event={"ID":"3dcb99e5-a045-4433-a3dc-2851abcba37a","Type":"ContainerStarted","Data":"1e904abd9e73d0d82cdf161fb8052aacb26650d5e0b350f50537a0a34c025b7c"}
Jul 27 19:40:09.140847 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.140211    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc" event={"ID":"3dcb99e5-a045-4433-a3dc-2851abcba37a","Type":"ContainerStarted","Data":"83c998651be8c7fb187c37508cd3e7bab52fff7d98c072fefd9913f9b2291ddb"}
Jul 27 19:40:09.140847 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.140257    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc" event={"ID":"3dcb99e5-a045-4433-a3dc-2851abcba37a","Type":"ContainerStarted","Data":"696c4a6317a810e7247edbf8630b6c49794ab2f03181064024447df159d91cb4"}
Jul 27 19:40:09.140847 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.140303    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc" event={"ID":"3dcb99e5-a045-4433-a3dc-2851abcba37a","Type":"ContainerStarted","Data":"cfc5710933950f764999ffc2a0bf746ca2bdddeb16d976be12e6457c474e4649"}
Jul 27 19:40:09.141292 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.141254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-42l2f" event={"ID":"63bd477d-eac4-4ca6-b5d3-4734ba278992","Type":"ContainerStarted","Data":"434b0a8ad89981f53b32bf62b8ca71748c652a8f0598903190b8ba29d6fa8c86"}
Jul 27 19:40:09.142678 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.142635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x5d7t" event={"ID":"af41025b-248c-4634-8c46-68d227fa3478","Type":"ContainerStarted","Data":"64be393538864ed1851fcb8e8047c4871fedc7a6e6bd5497238de799c7685ebf"}
Jul 27 19:40:09.144227 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.144202    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gm5lr" event={"ID":"e3bc4cbd-3f86-4404-a4f2-10c765d2bd32","Type":"ContainerStarted","Data":"48a0b56a1b2e68ec06b4607b17908db8faa0f0ea9c95788aa53b8f411fa75d51"}
Jul 27 19:40:09.145733 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.145707    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf" event={"ID":"2d58052c-f824-4f53-8459-6c6f16e4bac9","Type":"ContainerStarted","Data":"03986554b52010fe7497c54628970d914fa9dde5f23e6bdf80d3f9b03bfb7cb5"}
Jul 27 19:40:09.147369 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.147348    2562 generic.go:356] "Generic (PLEG): container finished" podID="8413e456-000f-457a-9ad6-a337819268cc" containerID="aa3c8c5f3de74c36f7900704631f62281fc416f39486f77b407b99147fe14f21" exitCode=0
Jul 27 19:40:09.147501 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.147430    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mwwb" event={"ID":"8413e456-000f-457a-9ad6-a337819268cc","Type":"ContainerDied","Data":"aa3c8c5f3de74c36f7900704631f62281fc416f39486f77b407b99147fe14f21"}
Jul 27 19:40:09.149237 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.149215    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8qf2p" event={"ID":"d50693b0-f760-4b28-8b31-514885f144ab","Type":"ContainerStarted","Data":"9867bdd691a93724aeb4675490d4f588be55fbef98f7b13adca9aa8570cea069"}
Jul 27 19:40:09.150865 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.150842    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bglxw" event={"ID":"00b00751-0c4b-431b-8699-6aad21590c67","Type":"ContainerStarted","Data":"f2f0fd7484452b2e163ec05ea5268bbeaad6e072131960dc46ccf21c8cc48fa2"}
Jul 27 19:40:09.160852 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.160808    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-42l2f" podStartSLOduration=3.873132513 podStartE2EDuration="23.160768034s" totalImagesPullingTime="19.287635521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:09.1606146 +0000 UTC m=+23.712531031" watchObservedRunningTime="2026-07-27 19:40:09.160768034 +0000 UTC m=+23.712684449"
Jul 27 19:40:09.161926 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.161890    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-140.ec2.internal" podStartSLOduration=22.16187405 podStartE2EDuration="22.16187405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:39:51.117600787 +0000 UTC m=+5.669517210" watchObservedRunningTime="2026-07-27 19:40:09.16187405 +0000 UTC m=+23.713790469"
Jul 27 19:40:09.213783 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.213689    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x5d7t" podStartSLOduration=4.030785441 podStartE2EDuration="23.21367355s" totalImagesPullingTime="19.182888109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:09.200154206 +0000 UTC m=+23.752070623" watchObservedRunningTime="2026-07-27 19:40:09.21367355 +0000 UTC m=+23.765589962"
Jul 27 19:40:09.228300 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.228250    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bglxw" podStartSLOduration=3.031471037 podStartE2EDuration="22.228237038s" totalImagesPullingTime="19.196766001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:09.214120391 +0000 UTC m=+23.766036809" watchObservedRunningTime="2026-07-27 19:40:09.228237038 +0000 UTC m=+23.780153456"
Jul 27 19:40:09.228456 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.228319    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8qf2p" podStartSLOduration=4.037257263 podStartE2EDuration="23.228315408s" totalImagesPullingTime="19.191058145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:09.227769234 +0000 UTC m=+23.779685653" watchObservedRunningTime="2026-07-27 19:40:09.228315408 +0000 UTC m=+23.780231825"
Jul 27 19:40:09.761174 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.761148    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 19:40:09.944630 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.944541    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T19:40:09.761169698Z","UUID":"0eed6146-8fe1-4e3d-8d9c-333d9595bed6","Handler":null,"Name":"","Endpoint":""}
Jul 27 19:40:09.946063 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.946041    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 19:40:09.946172 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.946075    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 19:40:09.985603 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.985575    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:09.985740 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.985579    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:09.985740 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:09.985672    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:40:09.985740 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:09.985678    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:09.985866 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:09.985782    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:40:09.985902 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:09.985874    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:40:10.155699 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:10.155273    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xh555" event={"ID":"ec6c229c-8276-42c2-9001-54332cc9da33","Type":"ContainerStarted","Data":"e80305cf98061b79cd78227e0c60c01a1d8b3fdeb80be01d874aa25794497621"}
Jul 27 19:40:10.158550 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:10.158427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf" event={"ID":"2d58052c-f824-4f53-8459-6c6f16e4bac9","Type":"ContainerStarted","Data":"5cb0ca17529d51012b901c043276f628659f240937dff90dd98eaa7ca1df6c5c"}
Jul 27 19:40:10.168996 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:10.168940    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gm5lr" podStartSLOduration=3.942242915 podStartE2EDuration="23.168919915s" totalImagesPullingTime="19.226677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:09.243240955 +0000 UTC m=+23.795157373" watchObservedRunningTime="2026-07-27 19:40:10.168919915 +0000 UTC m=+24.720836337"
Jul 27 19:40:10.169523 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:10.169496    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xh555" podStartSLOduration=3.950028646 podStartE2EDuration="23.169466663s" totalImagesPullingTime="19.219438017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:10.169066041 +0000 UTC m=+24.720982451" watchObservedRunningTime="2026-07-27 19:40:10.169466663 +0000 UTC m=+24.721383097"
Jul 27 19:40:10.244695 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:10.244610    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x5d7t"
Jul 27 19:40:10.245302 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:10.245278    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x5d7t"
Jul 27 19:40:11.163941 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:11.163903    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf" event={"ID":"2d58052c-f824-4f53-8459-6c6f16e4bac9","Type":"ContainerStarted","Data":"1dbce4074f68896f0678840087d4433d14121df308c58f92c4bcabf7ce17fe21"}
Jul 27 19:40:11.167669 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:11.167634    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc" event={"ID":"3dcb99e5-a045-4433-a3dc-2851abcba37a","Type":"ContainerStarted","Data":"cb9a95c27d65b6955d7d651e7e5af31492b9244ad5737a7f43435d95066954ec"}
Jul 27 19:40:11.167955 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:11.167932    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x5d7t"
Jul 27 19:40:11.168375 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:11.168361    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x5d7t"
Jul 27 19:40:11.182884 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:11.182840    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bnrrf" podStartSLOduration=2.848448099 podStartE2EDuration="24.182825609s" totalImagesPullingTime="21.33437751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:11.182785625 +0000 UTC m=+25.734702044" watchObservedRunningTime="2026-07-27 19:40:11.182825609 +0000 UTC m=+25.734742018"
Jul 27 19:40:11.984954 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:11.984924    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:11.985231 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:11.984924    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:11.985231 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:11.985043    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:40:11.985231 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:11.985074    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:11.985231 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:11.985166    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:40:11.985467 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:11.985259    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:40:13.987732 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:13.987704    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:13.988462 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:13.987714    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:13.988462 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:13.987817    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:40:13.988462 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:13.987714    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:13.988462 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:13.987879    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:40:13.988462 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:13.987951    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:40:14.176903 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:14.176820    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc" event={"ID":"3dcb99e5-a045-4433-a3dc-2851abcba37a","Type":"ContainerStarted","Data":"42a4dd9123b8ae3bfb16c2ac51332a8ff1437f752596e7b512aa79043651e8e9"}
Jul 27 19:40:14.177064 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:14.177048    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:40:14.178648 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:14.178627    2562 generic.go:356] "Generic (PLEG): container finished" podID="8413e456-000f-457a-9ad6-a337819268cc" containerID="0dd2db38c0ff7e459128fa10b80c07b8fd99eb4301cfccefdeffdb177f773fce" exitCode=0
Jul 27 19:40:14.178752 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:14.178658    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mwwb" event={"ID":"8413e456-000f-457a-9ad6-a337819268cc","Type":"ContainerDied","Data":"0dd2db38c0ff7e459128fa10b80c07b8fd99eb4301cfccefdeffdb177f773fce"}
Jul 27 19:40:14.193232 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:14.193212    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:40:14.204607 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:14.204566    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc" podStartSLOduration=7.969490863 podStartE2EDuration="27.204553649s" totalImagesPullingTime="19.235062786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:14.203964651 +0000 UTC m=+28.755881080" watchObservedRunningTime="2026-07-27 19:40:14.204553649 +0000 UTC m=+28.756470067"
Jul 27 19:40:15.181843 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:15.181685    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:40:15.182246 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:15.181886    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:40:15.199043 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:15.199015    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:40:15.631628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:15.631594    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8q4hl"]
Jul 27 19:40:15.631838 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:15.631726    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:15.631900 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:15.631840    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:40:15.635148 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:15.635124    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jbl6q"]
Jul 27 19:40:15.635260 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:15.635227    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:15.635322 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:15.635301    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:40:15.635905 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:15.635883    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z2n28"]
Jul 27 19:40:15.636020 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:15.636003    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:15.636133 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:15.636113    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:40:16.185182 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:16.185148    2562 generic.go:356] "Generic (PLEG): container finished" podID="8413e456-000f-457a-9ad6-a337819268cc" containerID="24440e3be9c7bc29b6d8b98781556b6b3ec3395a89447b4312d2b38efb525737" exitCode=0
Jul 27 19:40:16.185615 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:16.185191    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mwwb" event={"ID":"8413e456-000f-457a-9ad6-a337819268cc","Type":"ContainerDied","Data":"24440e3be9c7bc29b6d8b98781556b6b3ec3395a89447b4312d2b38efb525737"}
Jul 27 19:40:16.984925 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:16.984883    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:16.985107 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:16.985019    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:16.985107 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:16.985034    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:40:16.985107 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:16.985048    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:16.985235 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:16.985107    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:40:16.985235 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:16.985187    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:40:18.192125 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:18.191937    2562 generic.go:356] "Generic (PLEG): container finished" podID="8413e456-000f-457a-9ad6-a337819268cc" containerID="143718983d1d09224d27e0adebd6db00a0872e777176e197cfd782ba5f5fe244" exitCode=0
Jul 27 19:40:18.192616 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:18.192032    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mwwb" event={"ID":"8413e456-000f-457a-9ad6-a337819268cc","Type":"ContainerDied","Data":"143718983d1d09224d27e0adebd6db00a0872e777176e197cfd782ba5f5fe244"}
Jul 27 19:40:18.985668 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:18.985642    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:18.985834 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:18.985642    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:18.985834 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:18.985756    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:40:18.985834 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:18.985642    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:18.985985 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:18.985833    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:40:18.985985 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:18.985922    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:40:19.652312 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:19.652267    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:19.652803 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:19.652332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs\") pod \"network-metrics-daemon-z2n28\" (UID: \"cb8b05dc-fed1-4ced-b993-489c5da47ecf\") " pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:19.652803 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:19.652374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnn92\" (UniqueName: \"kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92\") pod \"network-check-target-8q4hl\" (UID: \"1ebc8908-1ce3-489f-9401-49714efbaf34\") " pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:19.652803 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:19.652422    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:40:19.652803 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:19.652473    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:40:19.652803 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:19.652512    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:40:19.652803 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:19.652526    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:40:19.652803 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:19.652536    2562 projected.go:196] Error preparing data for projected volume kube-api-access-pnn92 for pod openshift-network-diagnostics/network-check-target-8q4hl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:40:19.652803 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:19.652513    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret podName:003428df-67c2-4b3c-906a-260c8728b23a nodeName:}" failed. No retries permitted until 2026-07-27 19:40:51.652491927 +0000 UTC m=+66.204408342 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret") pod "global-pull-secret-syncer-jbl6q" (UID: "003428df-67c2-4b3c-906a-260c8728b23a") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:40:19.652803 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:19.652584    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs podName:cb8b05dc-fed1-4ced-b993-489c5da47ecf nodeName:}" failed. No retries permitted until 2026-07-27 19:40:51.652569267 +0000 UTC m=+66.204485669 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs") pod "network-metrics-daemon-z2n28" (UID: "cb8b05dc-fed1-4ced-b993-489c5da47ecf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:40:19.652803 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:19.652601    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92 podName:1ebc8908-1ce3-489f-9401-49714efbaf34 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:51.652591362 +0000 UTC m=+66.204507759 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnn92" (UniqueName: "kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92") pod "network-check-target-8q4hl" (UID: "1ebc8908-1ce3-489f-9401-49714efbaf34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:40:20.985301 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:20.985262    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:20.985886 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:20.985261    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:20.985886 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:20.985395    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2n28" podUID="cb8b05dc-fed1-4ced-b993-489c5da47ecf"
Jul 27 19:40:20.985886 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:20.985269    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:20.985886 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:20.985496    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8q4hl" podUID="1ebc8908-1ce3-489f-9401-49714efbaf34"
Jul 27 19:40:20.985886 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:20.985587    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jbl6q" podUID="003428df-67c2-4b3c-906a-260c8728b23a"
Jul 27 19:40:21.299903 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.299873    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-140.ec2.internal" event="NodeReady"
Jul 27 19:40:21.300084 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.300024    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 19:40:21.353144 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.352185    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58c95c44fd-krgrl"]
Jul 27 19:40:21.358261 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.358233    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pg89k"]
Jul 27 19:40:21.358403 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.358374    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.360553 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.360530    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4lj5s\""
Jul 27 19:40:21.361377 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.361354    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 19:40:21.361511 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.361451    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 19:40:21.361579 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.361514    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 19:40:21.361869 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.361849    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9vv5w"]
Jul 27 19:40:21.361980 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.361931    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pg89k"
Jul 27 19:40:21.365433 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.365413    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:21.365896 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.365817    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cqn6v\""
Jul 27 19:40:21.366015 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.365997    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 19:40:21.366162 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.366147    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 19:40:21.366350 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.366334    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 19:40:21.367152 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.366977    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 19:40:21.367247 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.367229    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 19:40:21.367295 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.367253    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 19:40:21.367336 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.367297    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tnv29\""
Jul 27 19:40:21.369703 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.369679    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pg89k"]
Jul 27 19:40:21.371320 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.371288    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9vv5w"]
Jul 27 19:40:21.371409 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.371338    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58c95c44fd-krgrl"]
Jul 27 19:40:21.467956 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.467922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.467956 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.467966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765ab079-566d-47ea-b202-f685ec0116e2-tmp-dir\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:21.468207 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.467996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c0c29fcc-6f35-4888-92a8-020814c281d1-ca-trust-extracted\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.468207 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.468021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-bound-sa-token\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.468207 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.468064    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbbzw\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-kube-api-access-wbbzw\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.468207 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.468134    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b38c083a-9531-4819-9439-7a151799aa81-cert\") pod \"ingress-canary-pg89k\" (UID: \"b38c083a-9531-4819-9439-7a151799aa81\") " pod="openshift-ingress-canary/ingress-canary-pg89k"
Jul 27 19:40:21.468207 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.468165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-certificates\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.468207 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.468180    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52g5l\" (UniqueName: \"kubernetes.io/projected/b38c083a-9531-4819-9439-7a151799aa81-kube-api-access-52g5l\") pod \"ingress-canary-pg89k\" (UID: \"b38c083a-9531-4819-9439-7a151799aa81\") " pod="openshift-ingress-canary/ingress-canary-pg89k"
Jul 27 19:40:21.468436 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.468221    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c0c29fcc-6f35-4888-92a8-020814c281d1-image-registry-private-configuration\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.468436 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.468272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:21.468436 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.468308    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znr2c\" (UniqueName: \"kubernetes.io/projected/765ab079-566d-47ea-b202-f685ec0116e2-kube-api-access-znr2c\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:21.468436 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.468341    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/765ab079-566d-47ea-b202-f685ec0116e2-config-volume\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:21.468436 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.468356    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0c29fcc-6f35-4888-92a8-020814c281d1-trusted-ca\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.468436 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.468396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c0c29fcc-6f35-4888-92a8-020814c281d1-installation-pull-secrets\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.569503 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.569404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.569503 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.569455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765ab079-566d-47ea-b202-f685ec0116e2-tmp-dir\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:21.569503 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.569498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c0c29fcc-6f35-4888-92a8-020814c281d1-ca-trust-extracted\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.569779 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.569522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-bound-sa-token\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.569779 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.569548    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbbzw\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-kube-api-access-wbbzw\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.569779 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.569573    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b38c083a-9531-4819-9439-7a151799aa81-cert\") pod \"ingress-canary-pg89k\" (UID: \"b38c083a-9531-4819-9439-7a151799aa81\") " pod="openshift-ingress-canary/ingress-canary-pg89k"
Jul 27 19:40:21.569779 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:21.569599    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:40:21.569779 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:21.569620    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58c95c44fd-krgrl: secret "image-registry-tls" not found
Jul 27 19:40:21.569779 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:21.569683    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls podName:c0c29fcc-6f35-4888-92a8-020814c281d1 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:22.069661392 +0000 UTC m=+36.621577807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls") pod "image-registry-58c95c44fd-krgrl" (UID: "c0c29fcc-6f35-4888-92a8-020814c281d1") : secret "image-registry-tls" not found
Jul 27 19:40:21.570073 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.569604    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-certificates\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.570073 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.569827    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52g5l\" (UniqueName: \"kubernetes.io/projected/b38c083a-9531-4819-9439-7a151799aa81-kube-api-access-52g5l\") pod \"ingress-canary-pg89k\" (UID: \"b38c083a-9531-4819-9439-7a151799aa81\") " pod="openshift-ingress-canary/ingress-canary-pg89k"
Jul 27 19:40:21.570073 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.569841    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765ab079-566d-47ea-b202-f685ec0116e2-tmp-dir\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:21.570073 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.569893    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c0c29fcc-6f35-4888-92a8-020814c281d1-image-registry-private-configuration\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.570073 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.569920    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c0c29fcc-6f35-4888-92a8-020814c281d1-ca-trust-extracted\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.570073 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.569937    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:21.570073 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.569975    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znr2c\" (UniqueName: \"kubernetes.io/projected/765ab079-566d-47ea-b202-f685ec0116e2-kube-api-access-znr2c\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:21.570073 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.570010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/765ab079-566d-47ea-b202-f685ec0116e2-config-volume\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:21.570073 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:21.570017    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:40:21.570073 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.570036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0c29fcc-6f35-4888-92a8-020814c281d1-trusted-ca\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.570073 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:21.570066    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls podName:765ab079-566d-47ea-b202-f685ec0116e2 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:22.070044398 +0000 UTC m=+36.621960798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls") pod "dns-default-9vv5w" (UID: "765ab079-566d-47ea-b202-f685ec0116e2") : secret "dns-default-metrics-tls" not found
Jul 27 19:40:21.570571 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.570122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c0c29fcc-6f35-4888-92a8-020814c281d1-installation-pull-secrets\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.570571 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.570171    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-certificates\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.570656 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.570600    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/765ab079-566d-47ea-b202-f685ec0116e2-config-volume\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:21.570895 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.570872    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0c29fcc-6f35-4888-92a8-020814c281d1-trusted-ca\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.575028 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.575005    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c0c29fcc-6f35-4888-92a8-020814c281d1-image-registry-private-configuration\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.575130 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.575028    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b38c083a-9531-4819-9439-7a151799aa81-cert\") pod \"ingress-canary-pg89k\" (UID: \"b38c083a-9531-4819-9439-7a151799aa81\") " pod="openshift-ingress-canary/ingress-canary-pg89k"
Jul 27 19:40:21.575130 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.575058    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c0c29fcc-6f35-4888-92a8-020814c281d1-installation-pull-secrets\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.585068 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.585042    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znr2c\" (UniqueName: \"kubernetes.io/projected/765ab079-566d-47ea-b202-f685ec0116e2-kube-api-access-znr2c\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:21.585196 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.585104    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52g5l\" (UniqueName: \"kubernetes.io/projected/b38c083a-9531-4819-9439-7a151799aa81-kube-api-access-52g5l\") pod \"ingress-canary-pg89k\" (UID: \"b38c083a-9531-4819-9439-7a151799aa81\") " pod="openshift-ingress-canary/ingress-canary-pg89k"
Jul 27 19:40:21.585196 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.585135    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbbzw\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-kube-api-access-wbbzw\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.587017 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.586996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-bound-sa-token\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:21.681817 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:21.681779    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pg89k"
Jul 27 19:40:22.074159 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:22.074122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:22.074744 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:22.074202    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:22.074744 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:22.074276    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:40:22.074744 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:22.074309    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:40:22.074744 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:22.074335    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58c95c44fd-krgrl: secret "image-registry-tls" not found
Jul 27 19:40:22.074744 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:22.074344    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls podName:765ab079-566d-47ea-b202-f685ec0116e2 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:23.074328188 +0000 UTC m=+37.626244585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls") pod "dns-default-9vv5w" (UID: "765ab079-566d-47ea-b202-f685ec0116e2") : secret "dns-default-metrics-tls" not found
Jul 27 19:40:22.074744 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:22.074395    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls podName:c0c29fcc-6f35-4888-92a8-020814c281d1 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:23.074380803 +0000 UTC m=+37.626297213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls") pod "image-registry-58c95c44fd-krgrl" (UID: "c0c29fcc-6f35-4888-92a8-020814c281d1") : secret "image-registry-tls" not found
Jul 27 19:40:22.183107 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:22.182624    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pg89k"]
Jul 27 19:40:22.192402 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:40:22.192368    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38c083a_9531_4819_9439_7a151799aa81.slice/crio-a77bed5a6996ebabd6f8edea6f4b135b9fe39d9e1d2d28d459e2aeff40cd2ac2 WatchSource:0}: Error finding container a77bed5a6996ebabd6f8edea6f4b135b9fe39d9e1d2d28d459e2aeff40cd2ac2: Status 404 returned error can't find the container with id a77bed5a6996ebabd6f8edea6f4b135b9fe39d9e1d2d28d459e2aeff40cd2ac2
Jul 27 19:40:22.202666 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:22.202582    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pg89k" event={"ID":"b38c083a-9531-4819-9439-7a151799aa81","Type":"ContainerStarted","Data":"a77bed5a6996ebabd6f8edea6f4b135b9fe39d9e1d2d28d459e2aeff40cd2ac2"}
Jul 27 19:40:22.985287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:22.985249    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:22.985287 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:22.985279    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:22.985542 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:22.985287    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:22.987749 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:22.987726    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 19:40:22.987749 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:22.987749    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 19:40:22.987893 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:22.987807    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 19:40:22.988676 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:22.988162    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4fnrz\""
Jul 27 19:40:22.988676 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:22.988349    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 19:40:22.988676 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:22.988550    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-klph9\""
Jul 27 19:40:23.081751 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:23.081713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:23.082328 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:23.081777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:23.082328 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:23.081880    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:40:23.082328 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:23.081895    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:40:23.082328 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:23.081912    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58c95c44fd-krgrl: secret "image-registry-tls" not found
Jul 27 19:40:23.082328 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:23.081956    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls podName:765ab079-566d-47ea-b202-f685ec0116e2 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:25.081933275 +0000 UTC m=+39.633849686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls") pod "dns-default-9vv5w" (UID: "765ab079-566d-47ea-b202-f685ec0116e2") : secret "dns-default-metrics-tls" not found
Jul 27 19:40:23.082328 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:23.082002    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls podName:c0c29fcc-6f35-4888-92a8-020814c281d1 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:25.081983744 +0000 UTC m=+39.633900158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls") pod "image-registry-58c95c44fd-krgrl" (UID: "c0c29fcc-6f35-4888-92a8-020814c281d1") : secret "image-registry-tls" not found
Jul 27 19:40:23.208950 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:23.208912    2562 generic.go:356] "Generic (PLEG): container finished" podID="8413e456-000f-457a-9ad6-a337819268cc" containerID="16c7c049b6bf115500aa10b45d558231bf7d62e66883b857457442320d4a18fd" exitCode=0
Jul 27 19:40:23.209111 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:23.208967    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mwwb" event={"ID":"8413e456-000f-457a-9ad6-a337819268cc","Type":"ContainerDied","Data":"16c7c049b6bf115500aa10b45d558231bf7d62e66883b857457442320d4a18fd"}
Jul 27 19:40:24.215197 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:24.215166    2562 generic.go:356] "Generic (PLEG): container finished" podID="8413e456-000f-457a-9ad6-a337819268cc" containerID="69156e0d55b33ed991c8f45628ddfd124cd25d9489ece8e61643b0e4b483d0ed" exitCode=0
Jul 27 19:40:24.215630 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:24.215219    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mwwb" event={"ID":"8413e456-000f-457a-9ad6-a337819268cc","Type":"ContainerDied","Data":"69156e0d55b33ed991c8f45628ddfd124cd25d9489ece8e61643b0e4b483d0ed"}
Jul 27 19:40:25.098729 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:25.098695    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:25.098908 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:25.098777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:25.098908 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:25.098888    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:40:25.098974 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:25.098944    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls podName:765ab079-566d-47ea-b202-f685ec0116e2 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:29.098931709 +0000 UTC m=+43.650848104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls") pod "dns-default-9vv5w" (UID: "765ab079-566d-47ea-b202-f685ec0116e2") : secret "dns-default-metrics-tls" not found
Jul 27 19:40:25.098974 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:25.098891    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:40:25.098974 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:25.098966    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58c95c44fd-krgrl: secret "image-registry-tls" not found
Jul 27 19:40:25.099072 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:25.099003    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls podName:c0c29fcc-6f35-4888-92a8-020814c281d1 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:29.098993756 +0000 UTC m=+43.650910155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls") pod "image-registry-58c95c44fd-krgrl" (UID: "c0c29fcc-6f35-4888-92a8-020814c281d1") : secret "image-registry-tls" not found
Jul 27 19:40:25.224530 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:25.224318    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7mwwb" event={"ID":"8413e456-000f-457a-9ad6-a337819268cc","Type":"ContainerStarted","Data":"69efa9a4f6fb23d68cff4c3f99cb850a30d21736181bb98178040f1e6fdf5e91"}
Jul 27 19:40:25.225658 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:25.225633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pg89k" event={"ID":"b38c083a-9531-4819-9439-7a151799aa81","Type":"ContainerStarted","Data":"7841de8aabf36fa6283f523b53ec7c261d987a2b74ceb52feb7204244867fc2c"}
Jul 27 19:40:25.246425 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:25.246378    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7mwwb" podStartSLOduration=9.421288697 podStartE2EDuration="39.246364049s" totalImagesPullingTime="29.825075352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:25.245607249 +0000 UTC m=+39.797523667" watchObservedRunningTime="2026-07-27 19:40:25.246364049 +0000 UTC m=+39.798280465"
Jul 27 19:40:25.260806 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:25.260755    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pg89k" podStartSLOduration=1.883864711 podStartE2EDuration="4.260739874s" totalImagesPullingTime="2.376875163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:25.260491822 +0000 UTC m=+39.812408236" watchObservedRunningTime="2026-07-27 19:40:25.260739874 +0000 UTC m=+39.812656313"
Jul 27 19:40:26.354082 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.354046    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-fh6px"]
Jul 27 19:40:26.384076 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.384041    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-fh6px"]
Jul 27 19:40:26.384222 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.384152    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-fh6px"
Jul 27 19:40:26.386260 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.386238    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 27 19:40:26.386386 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.386306    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 27 19:40:26.386602 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.386587    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4rqw6\""
Jul 27 19:40:26.508849 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.508820    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brqj9\" (UniqueName: \"kubernetes.io/projected/f052f509-6952-4c14-a833-cee46a6f8564-kube-api-access-brqj9\") pod \"migrator-6865c9967-fh6px\" (UID: \"f052f509-6952-4c14-a833-cee46a6f8564\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-fh6px"
Jul 27 19:40:26.575331 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.575305    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l46qm"]
Jul 27 19:40:26.595252 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.595193    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l46qm"]
Jul 27 19:40:26.595398 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.595312    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:26.597204 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.597181    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 19:40:26.597323 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.597249    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 19:40:26.597323 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.597279    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-csj86\""
Jul 27 19:40:26.597323 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.597258    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 19:40:26.597591 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.597577    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 19:40:26.610115 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.610062    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brqj9\" (UniqueName: \"kubernetes.io/projected/f052f509-6952-4c14-a833-cee46a6f8564-kube-api-access-brqj9\") pod \"migrator-6865c9967-fh6px\" (UID: \"f052f509-6952-4c14-a833-cee46a6f8564\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-fh6px"
Jul 27 19:40:26.621081 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.621053    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brqj9\" (UniqueName: \"kubernetes.io/projected/f052f509-6952-4c14-a833-cee46a6f8564-kube-api-access-brqj9\") pod \"migrator-6865c9967-fh6px\" (UID: \"f052f509-6952-4c14-a833-cee46a6f8564\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-fh6px"
Jul 27 19:40:26.692957 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.692919    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-fh6px"
Jul 27 19:40:26.711278 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.711248    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2wbt\" (UniqueName: \"kubernetes.io/projected/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-api-access-m2wbt\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:26.711457 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.711320    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-data-volume\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:26.711457 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.711396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-crio-socket\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:26.711457 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.711445    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:26.711611 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.711504    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:26.812886 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.812854    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-data-volume\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:26.813030 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.812898    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-crio-socket\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:26.813030 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.812934    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:26.813030 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.812961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:26.813030 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.813006    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2wbt\" (UniqueName: \"kubernetes.io/projected/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-api-access-m2wbt\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:26.813210 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.813161    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-data-volume\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:26.813314 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:26.813239    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:40:26.813314 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:26.813265    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:40:26.813314 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:26.813307    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm podName:625ce43b-c9a8-4a13-91e3-b6e014b59c1b nodeName:}" failed. No retries permitted until 2026-07-27 19:40:27.313288068 +0000 UTC m=+41.865204481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l46qm" (UID: "625ce43b-c9a8-4a13-91e3-b6e014b59c1b") : configmap "kube-rbac-proxy" not found
Jul 27 19:40:26.813418 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.813319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-crio-socket\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:26.813418 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:26.813326    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls podName:625ce43b-c9a8-4a13-91e3-b6e014b59c1b nodeName:}" failed. No retries permitted until 2026-07-27 19:40:27.313317472 +0000 UTC m=+41.865233868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l46qm" (UID: "625ce43b-c9a8-4a13-91e3-b6e014b59c1b") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:40:26.821628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.821596    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2wbt\" (UniqueName: \"kubernetes.io/projected/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-api-access-m2wbt\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:26.865108 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:26.865024    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-fh6px"]
Jul 27 19:40:26.868079 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:40:26.868024    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf052f509_6952_4c14_a833_cee46a6f8564.slice/crio-8010d642d22e6e458a35e8026ef56e5d7d5df74d4f0deb45d220210d47fbda48 WatchSource:0}: Error finding container 8010d642d22e6e458a35e8026ef56e5d7d5df74d4f0deb45d220210d47fbda48: Status 404 returned error can't find the container with id 8010d642d22e6e458a35e8026ef56e5d7d5df74d4f0deb45d220210d47fbda48
Jul 27 19:40:27.231056 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:27.231025    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-fh6px" event={"ID":"f052f509-6952-4c14-a833-cee46a6f8564","Type":"ContainerStarted","Data":"8010d642d22e6e458a35e8026ef56e5d7d5df74d4f0deb45d220210d47fbda48"}
Jul 27 19:40:27.318449 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:27.318404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:27.318449 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:27.318450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:27.318691 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:27.318555    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:40:27.318691 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:27.318613    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:40:27.318691 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:27.318622    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm podName:625ce43b-c9a8-4a13-91e3-b6e014b59c1b nodeName:}" failed. No retries permitted until 2026-07-27 19:40:28.318605605 +0000 UTC m=+42.870522000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l46qm" (UID: "625ce43b-c9a8-4a13-91e3-b6e014b59c1b") : configmap "kube-rbac-proxy" not found
Jul 27 19:40:27.318691 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:27.318652    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls podName:625ce43b-c9a8-4a13-91e3-b6e014b59c1b nodeName:}" failed. No retries permitted until 2026-07-27 19:40:28.318641099 +0000 UTC m=+42.870557495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l46qm" (UID: "625ce43b-c9a8-4a13-91e3-b6e014b59c1b") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:40:28.282582 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.282548    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"]
Jul 27 19:40:28.303409 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.303378    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"]
Jul 27 19:40:28.303574 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.303507    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"
Jul 27 19:40:28.305622 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.305583    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 27 19:40:28.305622 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.305588    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 27 19:40:28.305766 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.305713    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-gkkx5\""
Jul 27 19:40:28.326631 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.326606    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:28.326757 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.326646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:28.326757 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:28.326684    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:40:28.326757 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:28.326748    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm podName:625ce43b-c9a8-4a13-91e3-b6e014b59c1b nodeName:}" failed. No retries permitted until 2026-07-27 19:40:30.326731202 +0000 UTC m=+44.878647598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l46qm" (UID: "625ce43b-c9a8-4a13-91e3-b6e014b59c1b") : configmap "kube-rbac-proxy" not found
Jul 27 19:40:28.326867 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:28.326793    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:40:28.326867 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:28.326843    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls podName:625ce43b-c9a8-4a13-91e3-b6e014b59c1b nodeName:}" failed. No retries permitted until 2026-07-27 19:40:30.326830619 +0000 UTC m=+44.878747014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l46qm" (UID: "625ce43b-c9a8-4a13-91e3-b6e014b59c1b") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:40:28.427832 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.427791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hsv74\" (UID: \"7475366e-af8a-4306-b809-22f9239e51d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"
Jul 27 19:40:28.427832 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.427840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7475366e-af8a-4306-b809-22f9239e51d7-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-hsv74\" (UID: \"7475366e-af8a-4306-b809-22f9239e51d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"
Jul 27 19:40:28.528890 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.528852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hsv74\" (UID: \"7475366e-af8a-4306-b809-22f9239e51d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"
Jul 27 19:40:28.529045 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.528899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7475366e-af8a-4306-b809-22f9239e51d7-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-hsv74\" (UID: \"7475366e-af8a-4306-b809-22f9239e51d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"
Jul 27 19:40:28.529045 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:28.529004    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 19:40:28.529133 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:28.529071    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert podName:7475366e-af8a-4306-b809-22f9239e51d7 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:29.029055716 +0000 UTC m=+43.580972113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-hsv74" (UID: "7475366e-af8a-4306-b809-22f9239e51d7") : secret "networking-console-plugin-cert" not found
Jul 27 19:40:28.539885 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.539861    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7475366e-af8a-4306-b809-22f9239e51d7-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-hsv74\" (UID: \"7475366e-af8a-4306-b809-22f9239e51d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"
Jul 27 19:40:28.576796 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.576762    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-xl7l9"]
Jul 27 19:40:28.596017 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.595989    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-xl7l9"]
Jul 27 19:40:28.596125 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.596079    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-xl7l9"
Jul 27 19:40:28.599004 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.598966    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 27 19:40:28.599129 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.599020    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 27 19:40:28.599129 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.599097    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-m49xn\""
Jul 27 19:40:28.599294 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.599278    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 27 19:40:28.599589 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.599573    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 27 19:40:28.730740 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.730696    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fbf1ce56-92d5-467f-aea3-894138f0906c-signing-cabundle\") pod \"service-ca-658d7b7c78-xl7l9\" (UID: \"fbf1ce56-92d5-467f-aea3-894138f0906c\") " pod="openshift-service-ca/service-ca-658d7b7c78-xl7l9"
Jul 27 19:40:28.730902 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.730748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fbf1ce56-92d5-467f-aea3-894138f0906c-signing-key\") pod \"service-ca-658d7b7c78-xl7l9\" (UID: \"fbf1ce56-92d5-467f-aea3-894138f0906c\") " pod="openshift-service-ca/service-ca-658d7b7c78-xl7l9"
Jul 27 19:40:28.730902 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.730795    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dnd4\" (UniqueName: \"kubernetes.io/projected/fbf1ce56-92d5-467f-aea3-894138f0906c-kube-api-access-6dnd4\") pod \"service-ca-658d7b7c78-xl7l9\" (UID: \"fbf1ce56-92d5-467f-aea3-894138f0906c\") " pod="openshift-service-ca/service-ca-658d7b7c78-xl7l9"
Jul 27 19:40:28.831461 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.831374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fbf1ce56-92d5-467f-aea3-894138f0906c-signing-cabundle\") pod \"service-ca-658d7b7c78-xl7l9\" (UID: \"fbf1ce56-92d5-467f-aea3-894138f0906c\") " pod="openshift-service-ca/service-ca-658d7b7c78-xl7l9"
Jul 27 19:40:28.831461 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.831425    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fbf1ce56-92d5-467f-aea3-894138f0906c-signing-key\") pod \"service-ca-658d7b7c78-xl7l9\" (UID: \"fbf1ce56-92d5-467f-aea3-894138f0906c\") " pod="openshift-service-ca/service-ca-658d7b7c78-xl7l9"
Jul 27 19:40:28.831461 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.831451    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnd4\" (UniqueName: \"kubernetes.io/projected/fbf1ce56-92d5-467f-aea3-894138f0906c-kube-api-access-6dnd4\") pod \"service-ca-658d7b7c78-xl7l9\" (UID: \"fbf1ce56-92d5-467f-aea3-894138f0906c\") " pod="openshift-service-ca/service-ca-658d7b7c78-xl7l9"
Jul 27 19:40:28.832142 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.832121    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fbf1ce56-92d5-467f-aea3-894138f0906c-signing-cabundle\") pod \"service-ca-658d7b7c78-xl7l9\" (UID: \"fbf1ce56-92d5-467f-aea3-894138f0906c\") " pod="openshift-service-ca/service-ca-658d7b7c78-xl7l9"
Jul 27 19:40:28.833968 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.833947    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fbf1ce56-92d5-467f-aea3-894138f0906c-signing-key\") pod \"service-ca-658d7b7c78-xl7l9\" (UID: \"fbf1ce56-92d5-467f-aea3-894138f0906c\") " pod="openshift-service-ca/service-ca-658d7b7c78-xl7l9"
Jul 27 19:40:28.839634 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.839601    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dnd4\" (UniqueName: \"kubernetes.io/projected/fbf1ce56-92d5-467f-aea3-894138f0906c-kube-api-access-6dnd4\") pod \"service-ca-658d7b7c78-xl7l9\" (UID: \"fbf1ce56-92d5-467f-aea3-894138f0906c\") " pod="openshift-service-ca/service-ca-658d7b7c78-xl7l9"
Jul 27 19:40:28.906544 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.906510    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-xl7l9"
Jul 27 19:40:28.972534 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:28.972510    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bglxw_00b00751-0c4b-431b-8699-6aad21590c67/dns-node-resolver/0.log"
Jul 27 19:40:29.034108 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:29.034068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hsv74\" (UID: \"7475366e-af8a-4306-b809-22f9239e51d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"
Jul 27 19:40:29.034315 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:29.034297    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 19:40:29.034385 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:29.034370    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert podName:7475366e-af8a-4306-b809-22f9239e51d7 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:30.03434925 +0000 UTC m=+44.586265670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-hsv74" (UID: "7475366e-af8a-4306-b809-22f9239e51d7") : secret "networking-console-plugin-cert" not found
Jul 27 19:40:29.065260 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:29.065229    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-xl7l9"]
Jul 27 19:40:29.068387 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:40:29.068362    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf1ce56_92d5_467f_aea3_894138f0906c.slice/crio-e9fcb1e94817364a75f2af080f55c94ea79b3f60689d0e8f5ec7d6b932255b9e WatchSource:0}: Error finding container e9fcb1e94817364a75f2af080f55c94ea79b3f60689d0e8f5ec7d6b932255b9e: Status 404 returned error can't find the container with id e9fcb1e94817364a75f2af080f55c94ea79b3f60689d0e8f5ec7d6b932255b9e
Jul 27 19:40:29.135029 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:29.134999    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:29.135163 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:29.135119    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:29.135213 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:29.135159    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:40:29.135213 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:29.135176    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58c95c44fd-krgrl: secret "image-registry-tls" not found
Jul 27 19:40:29.135274 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:29.135226    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:40:29.135274 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:29.135237    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls podName:c0c29fcc-6f35-4888-92a8-020814c281d1 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:37.135221435 +0000 UTC m=+51.687137834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls") pod "image-registry-58c95c44fd-krgrl" (UID: "c0c29fcc-6f35-4888-92a8-020814c281d1") : secret "image-registry-tls" not found
Jul 27 19:40:29.135274 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:29.135264    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls podName:765ab079-566d-47ea-b202-f685ec0116e2 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:37.13525343 +0000 UTC m=+51.687169826 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls") pod "dns-default-9vv5w" (UID: "765ab079-566d-47ea-b202-f685ec0116e2") : secret "dns-default-metrics-tls" not found
Jul 27 19:40:29.237610 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:29.237571    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-fh6px" event={"ID":"f052f509-6952-4c14-a833-cee46a6f8564","Type":"ContainerStarted","Data":"3034f680615d2fee992541b4faeb666067570942d119bb955d75016390a1ca8d"}
Jul 27 19:40:29.237808 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:29.237631    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-fh6px" event={"ID":"f052f509-6952-4c14-a833-cee46a6f8564","Type":"ContainerStarted","Data":"671b3fb052987d42e41c6b8c7535742851dd57bcd3161c6c5b556b1a03a908fd"}
Jul 27 19:40:29.238780 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:29.238758    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-xl7l9" event={"ID":"fbf1ce56-92d5-467f-aea3-894138f0906c","Type":"ContainerStarted","Data":"e9fcb1e94817364a75f2af080f55c94ea79b3f60689d0e8f5ec7d6b932255b9e"}
Jul 27 19:40:29.258695 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:29.258622    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-fh6px" podStartSLOduration=1.6958380690000001 podStartE2EDuration="3.25860286s" totalImagesPullingTime="1.562764791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:29.257812059 +0000 UTC m=+43.809728477" watchObservedRunningTime="2026-07-27 19:40:29.25860286 +0000 UTC m=+43.810519278"
Jul 27 19:40:29.776468 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:29.776437    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8qf2p_d50693b0-f760-4b28-8b31-514885f144ab/node-ca/0.log"
Jul 27 19:40:30.040898 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:30.040823    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hsv74\" (UID: \"7475366e-af8a-4306-b809-22f9239e51d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"
Jul 27 19:40:30.041036 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:30.040969    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 19:40:30.041075 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:30.041042    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert podName:7475366e-af8a-4306-b809-22f9239e51d7 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:32.041016111 +0000 UTC m=+46.592932510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-hsv74" (UID: "7475366e-af8a-4306-b809-22f9239e51d7") : secret "networking-console-plugin-cert" not found
Jul 27 19:40:30.344164 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:30.344084    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:30.344164 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:30.344128    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:30.344343 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:30.344221    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:40:30.344343 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:30.344262    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:40:30.344343 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:30.344292    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm podName:625ce43b-c9a8-4a13-91e3-b6e014b59c1b nodeName:}" failed. No retries permitted until 2026-07-27 19:40:34.344275783 +0000 UTC m=+48.896192178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l46qm" (UID: "625ce43b-c9a8-4a13-91e3-b6e014b59c1b") : configmap "kube-rbac-proxy" not found
Jul 27 19:40:30.344343 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:30.344306    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls podName:625ce43b-c9a8-4a13-91e3-b6e014b59c1b nodeName:}" failed. No retries permitted until 2026-07-27 19:40:34.344299598 +0000 UTC m=+48.896215994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l46qm" (UID: "625ce43b-c9a8-4a13-91e3-b6e014b59c1b") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:40:30.572790 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:30.572756    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pg89k_b38c083a-9531-4819-9439-7a151799aa81/serve-healthcheck-canary/0.log"
Jul 27 19:40:30.971755 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:30.971730    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-fh6px_f052f509-6952-4c14-a833-cee46a6f8564/migrator/0.log"
Jul 27 19:40:31.171053 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:31.171024    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-fh6px_f052f509-6952-4c14-a833-cee46a6f8564/graceful-termination/0.log"
Jul 27 19:40:32.056217 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:32.056182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hsv74\" (UID: \"7475366e-af8a-4306-b809-22f9239e51d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"
Jul 27 19:40:32.056736 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:32.056340    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 19:40:32.056736 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:32.056407    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert podName:7475366e-af8a-4306-b809-22f9239e51d7 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:36.056388959 +0000 UTC m=+50.608305355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-hsv74" (UID: "7475366e-af8a-4306-b809-22f9239e51d7") : secret "networking-console-plugin-cert" not found
Jul 27 19:40:32.247641 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:32.247606    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-xl7l9" event={"ID":"fbf1ce56-92d5-467f-aea3-894138f0906c","Type":"ContainerStarted","Data":"205008d1f7a535c753c0d72e3de325b2568143d08706679afa3e3cfd454fccc5"}
Jul 27 19:40:32.262021 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:32.261978    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-xl7l9" podStartSLOduration=2.089597433 podStartE2EDuration="4.261966233s" totalImagesPullingTime="2.1723688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:32.261035674 +0000 UTC m=+46.812952092" watchObservedRunningTime="2026-07-27 19:40:32.261966233 +0000 UTC m=+46.813882727"
Jul 27 19:40:34.372638 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:34.372602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:34.372638 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:34.372650    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:34.373055 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:34.372733    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:40:34.373055 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:34.372809    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm podName:625ce43b-c9a8-4a13-91e3-b6e014b59c1b nodeName:}" failed. No retries permitted until 2026-07-27 19:40:42.372790366 +0000 UTC m=+56.924706763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l46qm" (UID: "625ce43b-c9a8-4a13-91e3-b6e014b59c1b") : configmap "kube-rbac-proxy" not found
Jul 27 19:40:34.373055 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:34.372828    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:40:34.373055 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:34.372874    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls podName:625ce43b-c9a8-4a13-91e3-b6e014b59c1b nodeName:}" failed. No retries permitted until 2026-07-27 19:40:42.372861776 +0000 UTC m=+56.924778172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l46qm" (UID: "625ce43b-c9a8-4a13-91e3-b6e014b59c1b") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:40:36.087462 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:36.087425    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hsv74\" (UID: \"7475366e-af8a-4306-b809-22f9239e51d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"
Jul 27 19:40:36.087838 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:36.087587    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 19:40:36.087838 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:36.087652    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert podName:7475366e-af8a-4306-b809-22f9239e51d7 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:44.087636267 +0000 UTC m=+58.639552667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-hsv74" (UID: "7475366e-af8a-4306-b809-22f9239e51d7") : secret "networking-console-plugin-cert" not found
Jul 27 19:40:37.195354 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:37.195322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:37.195742 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:37.195378    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:37.195742 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:37.195515    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:40:37.195742 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:37.195525    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58c95c44fd-krgrl: secret "image-registry-tls" not found
Jul 27 19:40:37.195742 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:37.195576    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls podName:c0c29fcc-6f35-4888-92a8-020814c281d1 nodeName:}" failed. No retries permitted until 2026-07-27 19:40:53.19556014 +0000 UTC m=+67.747476554 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls") pod "image-registry-58c95c44fd-krgrl" (UID: "c0c29fcc-6f35-4888-92a8-020814c281d1") : secret "image-registry-tls" not found
Jul 27 19:40:37.197926 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:37.197903    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/765ab079-566d-47ea-b202-f685ec0116e2-metrics-tls\") pod \"dns-default-9vv5w\" (UID: \"765ab079-566d-47ea-b202-f685ec0116e2\") " pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:37.288824 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:37.288789    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:37.428019 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:37.427963    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9vv5w"]
Jul 27 19:40:37.431551 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:40:37.431507    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765ab079_566d_47ea_b202_f685ec0116e2.slice/crio-a04dc8f7d8dcae0bf50f83fd471fea2859eef2a4ffd50f0e3fa9000c76c8fc77 WatchSource:0}: Error finding container a04dc8f7d8dcae0bf50f83fd471fea2859eef2a4ffd50f0e3fa9000c76c8fc77: Status 404 returned error can't find the container with id a04dc8f7d8dcae0bf50f83fd471fea2859eef2a4ffd50f0e3fa9000c76c8fc77
Jul 27 19:40:38.265424 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:38.265378    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9vv5w" event={"ID":"765ab079-566d-47ea-b202-f685ec0116e2","Type":"ContainerStarted","Data":"a04dc8f7d8dcae0bf50f83fd471fea2859eef2a4ffd50f0e3fa9000c76c8fc77"}
Jul 27 19:40:40.272536 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:40.272471    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9vv5w" event={"ID":"765ab079-566d-47ea-b202-f685ec0116e2","Type":"ContainerStarted","Data":"ec5b08547fb218b908cfb542e4e8ccd58a21a5b0b1f2e732daf774d1f23339ba"}
Jul 27 19:40:40.272536 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:40.272554    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9vv5w" event={"ID":"765ab079-566d-47ea-b202-f685ec0116e2","Type":"ContainerStarted","Data":"7a8caedcc80c789392cb087ae3eef2a1827dc93356608d39f86461cf58ffda8f"}
Jul 27 19:40:40.272972 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:40.272615    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:40.291079 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:40.291033    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9vv5w" podStartSLOduration=16.974113508 podStartE2EDuration="19.29101808s" totalImagesPullingTime="2.316904572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:40.290825835 +0000 UTC m=+54.842742253" watchObservedRunningTime="2026-07-27 19:40:40.29101808 +0000 UTC m=+54.842934497"
Jul 27 19:40:42.443413 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:42.443374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:42.443413 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:42.443415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:42.443854 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:42.443543    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:40:42.443854 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:42.443594    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls podName:625ce43b-c9a8-4a13-91e3-b6e014b59c1b nodeName:}" failed. No retries permitted until 2026-07-27 19:40:58.44358074 +0000 UTC m=+72.995497136 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l46qm" (UID: "625ce43b-c9a8-4a13-91e3-b6e014b59c1b") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:40:42.443932 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:42.443910    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:44.158730 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:44.158688    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hsv74\" (UID: \"7475366e-af8a-4306-b809-22f9239e51d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"
Jul 27 19:40:44.161162 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:44.161134    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7475366e-af8a-4306-b809-22f9239e51d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hsv74\" (UID: \"7475366e-af8a-4306-b809-22f9239e51d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"
Jul 27 19:40:44.211776 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:44.211746    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"
Jul 27 19:40:44.333679 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:44.333646    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-hsv74"]
Jul 27 19:40:44.336634 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:40:44.336605    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7475366e_af8a_4306_b809_22f9239e51d7.slice/crio-055ffa8a42c63eb62aaa7be2fa81e6a9bc741033f32df9b9be1417226f6156bc WatchSource:0}: Error finding container 055ffa8a42c63eb62aaa7be2fa81e6a9bc741033f32df9b9be1417226f6156bc: Status 404 returned error can't find the container with id 055ffa8a42c63eb62aaa7be2fa81e6a9bc741033f32df9b9be1417226f6156bc
Jul 27 19:40:45.289183 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:45.289144    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74" event={"ID":"7475366e-af8a-4306-b809-22f9239e51d7","Type":"ContainerStarted","Data":"055ffa8a42c63eb62aaa7be2fa81e6a9bc741033f32df9b9be1417226f6156bc"}
Jul 27 19:40:46.293670 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:46.293635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74" event={"ID":"7475366e-af8a-4306-b809-22f9239e51d7","Type":"ContainerStarted","Data":"d57cdbeb7e624977d7e1f8a425faebd6b8efb9b04ed7b812279188348477987f"}
Jul 27 19:40:46.309176 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:46.309132    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-hsv74" podStartSLOduration=17.267908761 podStartE2EDuration="18.309118582s" totalImagesPullingTime="1.041209821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:46.308574811 +0000 UTC m=+60.860491230" watchObservedRunningTime="2026-07-27 19:40:46.309118582 +0000 UTC m=+60.861035020"
Jul 27 19:40:47.202070 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:47.202035    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8bnpc"
Jul 27 19:40:50.278284 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:50.278254    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9vv5w"
Jul 27 19:40:51.715935 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.715894    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:51.715935 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.715942    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs\") pod \"network-metrics-daemon-z2n28\" (UID: \"cb8b05dc-fed1-4ced-b993-489c5da47ecf\") " pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:51.716426 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.715966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnn92\" (UniqueName: \"kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92\") pod \"network-check-target-8q4hl\" (UID: \"1ebc8908-1ce3-489f-9401-49714efbaf34\") " pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:51.718272 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.718255    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 19:40:51.718738 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.718723    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 19:40:51.718790 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.718739    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 19:40:51.728358 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.728334    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 19:40:51.728508 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.728412    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8b05dc-fed1-4ced-b993-489c5da47ecf-metrics-certs\") pod \"network-metrics-daemon-z2n28\" (UID: \"cb8b05dc-fed1-4ced-b993-489c5da47ecf\") " pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:51.728508 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.728431    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/003428df-67c2-4b3c-906a-260c8728b23a-original-pull-secret\") pod \"global-pull-secret-syncer-jbl6q\" (UID: \"003428df-67c2-4b3c-906a-260c8728b23a\") " pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:51.739384 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.739360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnn92\" (UniqueName: \"kubernetes.io/projected/1ebc8908-1ce3-489f-9401-49714efbaf34-kube-api-access-pnn92\") pod \"network-check-target-8q4hl\" (UID: \"1ebc8908-1ce3-489f-9401-49714efbaf34\") " pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:51.799765 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.799736    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-klph9\""
Jul 27 19:40:51.804228 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.804201    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jbl6q"
Jul 27 19:40:51.807963 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.807945    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:51.812789 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.812768    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4fnrz\""
Jul 27 19:40:51.820758 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.820737    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2n28"
Jul 27 19:40:51.954986 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.954951    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jbl6q"]
Jul 27 19:40:51.968210 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.968160    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8q4hl"]
Jul 27 19:40:51.971754 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:40:51.971729    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebc8908_1ce3_489f_9401_49714efbaf34.slice/crio-3c3e353aa795105b2db1baedcf61eecce057338a2a6ae5d48756a26136dc4194 WatchSource:0}: Error finding container 3c3e353aa795105b2db1baedcf61eecce057338a2a6ae5d48756a26136dc4194: Status 404 returned error can't find the container with id 3c3e353aa795105b2db1baedcf61eecce057338a2a6ae5d48756a26136dc4194
Jul 27 19:40:51.989066 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:51.989045    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z2n28"]
Jul 27 19:40:51.989825 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:40:51.989797    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8b05dc_fed1_4ced_b993_489c5da47ecf.slice/crio-12807ccf62ddb9a73cfa9a038a9b54157266476128098d151a152b1774b34929 WatchSource:0}: Error finding container 12807ccf62ddb9a73cfa9a038a9b54157266476128098d151a152b1774b34929: Status 404 returned error can't find the container with id 12807ccf62ddb9a73cfa9a038a9b54157266476128098d151a152b1774b34929
Jul 27 19:40:52.311758 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.311668    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8q4hl" event={"ID":"1ebc8908-1ce3-489f-9401-49714efbaf34","Type":"ContainerStarted","Data":"3c3e353aa795105b2db1baedcf61eecce057338a2a6ae5d48756a26136dc4194"}
Jul 27 19:40:52.312803 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.312772    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jbl6q" event={"ID":"003428df-67c2-4b3c-906a-260c8728b23a","Type":"ContainerStarted","Data":"aec7eeebe8af8b923db6f69da4e80b8a47aed7fee235c5bc03cead97765a6f6a"}
Jul 27 19:40:52.313762 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.313740    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z2n28" event={"ID":"cb8b05dc-fed1-4ced-b993-489c5da47ecf","Type":"ContainerStarted","Data":"12807ccf62ddb9a73cfa9a038a9b54157266476128098d151a152b1774b34929"}
Jul 27 19:40:52.684745 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.683536    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-w6bsx"]
Jul 27 19:40:52.687992 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.687033    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-w6bsx"
Jul 27 19:40:52.691472 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.691446    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 19:40:52.691732 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.691714    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mwzms\""
Jul 27 19:40:52.691949 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.691936    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 19:40:52.709451 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.707586    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-w6bsx"]
Jul 27 19:40:52.709451 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.707686    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58c95c44fd-krgrl"]
Jul 27 19:40:52.709451 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:40:52.708026    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-58c95c44fd-krgrl" podUID="c0c29fcc-6f35-4888-92a8-020814c281d1"
Jul 27 19:40:52.824801 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.824763    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2td4k\" (UniqueName: \"kubernetes.io/projected/db3b9846-fa9a-40d0-b2fa-031c0c6212b2-kube-api-access-2td4k\") pod \"downloads-7c97dc4555-w6bsx\" (UID: \"db3b9846-fa9a-40d0-b2fa-031c0c6212b2\") " pod="openshift-console/downloads-7c97dc4555-w6bsx"
Jul 27 19:40:52.832793 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.832760    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78f95f545c-r7f6p"]
Jul 27 19:40:52.836080 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.836018    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:52.855598 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.855536    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78f95f545c-r7f6p"]
Jul 27 19:40:52.925804 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.925755    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/930cfab4-26cf-4c8d-84d8-f87825942497-installation-pull-secrets\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:52.926529 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.926059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/930cfab4-26cf-4c8d-84d8-f87825942497-trusted-ca\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:52.926529 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.926107    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/930cfab4-26cf-4c8d-84d8-f87825942497-registry-certificates\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:52.926529 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.926143    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/930cfab4-26cf-4c8d-84d8-f87825942497-image-registry-private-configuration\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:52.926529 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.926169    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkf5k\" (UniqueName: \"kubernetes.io/projected/930cfab4-26cf-4c8d-84d8-f87825942497-kube-api-access-dkf5k\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:52.926529 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.926218    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/930cfab4-26cf-4c8d-84d8-f87825942497-registry-tls\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:52.926529 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.926242    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/930cfab4-26cf-4c8d-84d8-f87825942497-ca-trust-extracted\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:52.926529 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.926296    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2td4k\" (UniqueName: \"kubernetes.io/projected/db3b9846-fa9a-40d0-b2fa-031c0c6212b2-kube-api-access-2td4k\") pod \"downloads-7c97dc4555-w6bsx\" (UID: \"db3b9846-fa9a-40d0-b2fa-031c0c6212b2\") " pod="openshift-console/downloads-7c97dc4555-w6bsx"
Jul 27 19:40:52.926529 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.926359    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/930cfab4-26cf-4c8d-84d8-f87825942497-bound-sa-token\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:52.944322 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:52.944249    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2td4k\" (UniqueName: \"kubernetes.io/projected/db3b9846-fa9a-40d0-b2fa-031c0c6212b2-kube-api-access-2td4k\") pod \"downloads-7c97dc4555-w6bsx\" (UID: \"db3b9846-fa9a-40d0-b2fa-031c0c6212b2\") " pod="openshift-console/downloads-7c97dc4555-w6bsx"
Jul 27 19:40:53.002142 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.002103    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-w6bsx"
Jul 27 19:40:53.027599 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.027560    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/930cfab4-26cf-4c8d-84d8-f87825942497-trusted-ca\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.027761 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.027606    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/930cfab4-26cf-4c8d-84d8-f87825942497-registry-certificates\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.027761 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.027643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/930cfab4-26cf-4c8d-84d8-f87825942497-image-registry-private-configuration\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.027761 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.027670    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkf5k\" (UniqueName: \"kubernetes.io/projected/930cfab4-26cf-4c8d-84d8-f87825942497-kube-api-access-dkf5k\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.027761 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.027712    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/930cfab4-26cf-4c8d-84d8-f87825942497-registry-tls\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.027761 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.027735    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/930cfab4-26cf-4c8d-84d8-f87825942497-ca-trust-extracted\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.028023 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.027789    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/930cfab4-26cf-4c8d-84d8-f87825942497-bound-sa-token\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.028023 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.027809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/930cfab4-26cf-4c8d-84d8-f87825942497-installation-pull-secrets\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.029507 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.028571    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/930cfab4-26cf-4c8d-84d8-f87825942497-ca-trust-extracted\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.029507 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.029237    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/930cfab4-26cf-4c8d-84d8-f87825942497-registry-certificates\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.029507 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.029440    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/930cfab4-26cf-4c8d-84d8-f87825942497-trusted-ca\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.031459 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.031412    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/930cfab4-26cf-4c8d-84d8-f87825942497-image-registry-private-configuration\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.031585 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.031458    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/930cfab4-26cf-4c8d-84d8-f87825942497-installation-pull-secrets\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.031927 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.031903    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/930cfab4-26cf-4c8d-84d8-f87825942497-registry-tls\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.039852 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.039828    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkf5k\" (UniqueName: \"kubernetes.io/projected/930cfab4-26cf-4c8d-84d8-f87825942497-kube-api-access-dkf5k\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.041206 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.041183    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/930cfab4-26cf-4c8d-84d8-f87825942497-bound-sa-token\") pod \"image-registry-78f95f545c-r7f6p\" (UID: \"930cfab4-26cf-4c8d-84d8-f87825942497\") " pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.150753 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.150360    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4lj5s\""
Jul 27 19:40:53.160021 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.158805    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:53.230655 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.230623    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:53.234225 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.234182    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls\") pod \"image-registry-58c95c44fd-krgrl\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") " pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:53.234733 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.234710    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-w6bsx"]
Jul 27 19:40:53.243674 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:40:53.243053    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3b9846_fa9a_40d0_b2fa_031c0c6212b2.slice/crio-59565f546ae5209c1b75348d6eb592973bb83ed7326be5f0efade49c1a3314ba WatchSource:0}: Error finding container 59565f546ae5209c1b75348d6eb592973bb83ed7326be5f0efade49c1a3314ba: Status 404 returned error can't find the container with id 59565f546ae5209c1b75348d6eb592973bb83ed7326be5f0efade49c1a3314ba
Jul 27 19:40:53.321730 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.321697    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z2n28" event={"ID":"cb8b05dc-fed1-4ced-b993-489c5da47ecf","Type":"ContainerStarted","Data":"a73dba726fd2adb3c15429335a6c1e130e4e4c2fd0d9910cea6150b8cbadc57e"}
Jul 27 19:40:53.323372 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.323320    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:53.323596 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.323538    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-w6bsx" event={"ID":"db3b9846-fa9a-40d0-b2fa-031c0c6212b2","Type":"ContainerStarted","Data":"59565f546ae5209c1b75348d6eb592973bb83ed7326be5f0efade49c1a3314ba"}
Jul 27 19:40:53.344472 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.344320    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:53.361132 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.361102    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78f95f545c-r7f6p"]
Jul 27 19:40:53.361855 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:40:53.361831    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930cfab4_26cf_4c8d_84d8_f87825942497.slice/crio-c039f494f16c7e071073497dada4c1d1b4d52da77de5946815a59a9d092f91c8 WatchSource:0}: Error finding container c039f494f16c7e071073497dada4c1d1b4d52da77de5946815a59a9d092f91c8: Status 404 returned error can't find the container with id c039f494f16c7e071073497dada4c1d1b4d52da77de5946815a59a9d092f91c8
Jul 27 19:40:53.432446 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.432372    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-certificates\") pod \"c0c29fcc-6f35-4888-92a8-020814c281d1\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") "
Jul 27 19:40:53.432446 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.432419    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0c29fcc-6f35-4888-92a8-020814c281d1-trusted-ca\") pod \"c0c29fcc-6f35-4888-92a8-020814c281d1\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") "
Jul 27 19:40:53.432446 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.432458    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c0c29fcc-6f35-4888-92a8-020814c281d1-image-registry-private-configuration\") pod \"c0c29fcc-6f35-4888-92a8-020814c281d1\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") "
Jul 27 19:40:53.432723 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.432514    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c0c29fcc-6f35-4888-92a8-020814c281d1-installation-pull-secrets\") pod \"c0c29fcc-6f35-4888-92a8-020814c281d1\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") "
Jul 27 19:40:53.432723 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.432549    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c0c29fcc-6f35-4888-92a8-020814c281d1-ca-trust-extracted\") pod \"c0c29fcc-6f35-4888-92a8-020814c281d1\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") "
Jul 27 19:40:53.432723 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.432596    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbbzw\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-kube-api-access-wbbzw\") pod \"c0c29fcc-6f35-4888-92a8-020814c281d1\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") "
Jul 27 19:40:53.432723 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.432622    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls\") pod \"c0c29fcc-6f35-4888-92a8-020814c281d1\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") "
Jul 27 19:40:53.432723 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.432654    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-bound-sa-token\") pod \"c0c29fcc-6f35-4888-92a8-020814c281d1\" (UID: \"c0c29fcc-6f35-4888-92a8-020814c281d1\") "
Jul 27 19:40:53.433390 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.433089    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c0c29fcc-6f35-4888-92a8-020814c281d1" (UID: "c0c29fcc-6f35-4888-92a8-020814c281d1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:40:53.433390 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.433355    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c29fcc-6f35-4888-92a8-020814c281d1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c0c29fcc-6f35-4888-92a8-020814c281d1" (UID: "c0c29fcc-6f35-4888-92a8-020814c281d1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:40:53.433623 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.433409    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c29fcc-6f35-4888-92a8-020814c281d1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c0c29fcc-6f35-4888-92a8-020814c281d1" (UID: "c0c29fcc-6f35-4888-92a8-020814c281d1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:40:53.436003 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.435943    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c0c29fcc-6f35-4888-92a8-020814c281d1" (UID: "c0c29fcc-6f35-4888-92a8-020814c281d1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:40:53.436003 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.435944    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c29fcc-6f35-4888-92a8-020814c281d1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c0c29fcc-6f35-4888-92a8-020814c281d1" (UID: "c0c29fcc-6f35-4888-92a8-020814c281d1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:40:53.436003 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.435996    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c29fcc-6f35-4888-92a8-020814c281d1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c0c29fcc-6f35-4888-92a8-020814c281d1" (UID: "c0c29fcc-6f35-4888-92a8-020814c281d1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:40:53.436574 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.436412    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-kube-api-access-wbbzw" (OuterVolumeSpecName: "kube-api-access-wbbzw") pod "c0c29fcc-6f35-4888-92a8-020814c281d1" (UID: "c0c29fcc-6f35-4888-92a8-020814c281d1"). InnerVolumeSpecName "kube-api-access-wbbzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:40:53.436574 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.436531    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c0c29fcc-6f35-4888-92a8-020814c281d1" (UID: "c0c29fcc-6f35-4888-92a8-020814c281d1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:40:53.533413 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.533306    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-certificates\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:40:53.533413 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.533332    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0c29fcc-6f35-4888-92a8-020814c281d1-trusted-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:40:53.533413 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.533342    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c0c29fcc-6f35-4888-92a8-020814c281d1-image-registry-private-configuration\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:40:53.533413 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.533354    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c0c29fcc-6f35-4888-92a8-020814c281d1-installation-pull-secrets\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:40:53.533413 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.533367    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c0c29fcc-6f35-4888-92a8-020814c281d1-ca-trust-extracted\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:40:53.533413 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.533376    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbbzw\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-kube-api-access-wbbzw\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:40:53.533413 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.533385    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-registry-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:40:53.533413 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:53.533393    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0c29fcc-6f35-4888-92a8-020814c281d1-bound-sa-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:40:54.334226 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:54.334080    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z2n28" event={"ID":"cb8b05dc-fed1-4ced-b993-489c5da47ecf","Type":"ContainerStarted","Data":"0e276e72c80aaf9e9279dcc4b3f68655bad833e16e9169409d6a1ac91b3143af"}
Jul 27 19:40:54.337343 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:54.336904    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58c95c44fd-krgrl"
Jul 27 19:40:54.337343 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:54.337284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f95f545c-r7f6p" event={"ID":"930cfab4-26cf-4c8d-84d8-f87825942497","Type":"ContainerStarted","Data":"a83a6f4f214e4609f157d35316df8b36a40ac4e91d0de8d0b5c12570069dd317"}
Jul 27 19:40:54.337343 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:54.337335    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f95f545c-r7f6p" event={"ID":"930cfab4-26cf-4c8d-84d8-f87825942497","Type":"ContainerStarted","Data":"c039f494f16c7e071073497dada4c1d1b4d52da77de5946815a59a9d092f91c8"}
Jul 27 19:40:54.338417 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:54.337363    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:40:54.356438 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:54.356353    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z2n28" podStartSLOduration=67.231555301 podStartE2EDuration="1m8.356331691s" totalImagesPullingTime="1.12477639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:54.355969274 +0000 UTC m=+68.907885693" watchObservedRunningTime="2026-07-27 19:40:54.356331691 +0000 UTC m=+68.908248113"
Jul 27 19:40:54.390506 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:54.390448    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58c95c44fd-krgrl"]
Jul 27 19:40:54.393728 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:54.393698    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58c95c44fd-krgrl"]
Jul 27 19:40:55.990191 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:55.990144    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0c29fcc-6f35-4888-92a8-020814c281d1" path="/var/lib/kubelet/pods/c0c29fcc-6f35-4888-92a8-020814c281d1/volumes"
Jul 27 19:40:58.353362 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:58.353321    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jbl6q" event={"ID":"003428df-67c2-4b3c-906a-260c8728b23a","Type":"ContainerStarted","Data":"ad4b0e29d2d497a48730390982564e39777d1b497f9c5c34f616b0b968b4c15b"}
Jul 27 19:40:58.355023 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:58.354998    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8q4hl" event={"ID":"1ebc8908-1ce3-489f-9401-49714efbaf34","Type":"ContainerStarted","Data":"79a97a863512e62959df308870b5933987ac527bc84380e06e160b3643c3546e"}
Jul 27 19:40:58.355153 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:58.355140    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:40:58.369742 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:58.369679    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78f95f545c-r7f6p" podStartSLOduration=6.369659628 podStartE2EDuration="6.369659628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:54.416984289 +0000 UTC m=+68.968900708" watchObservedRunningTime="2026-07-27 19:40:58.369659628 +0000 UTC m=+72.921576043"
Jul 27 19:40:58.370164 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:58.370121    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jbl6q" podStartSLOduration=65.741866828 podStartE2EDuration="1m11.370112211s" totalImagesPullingTime="5.628245383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:58.368765431 +0000 UTC m=+72.920681850" watchObservedRunningTime="2026-07-27 19:40:58.370112211 +0000 UTC m=+72.922028629"
Jul 27 19:40:58.383125 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:58.382943    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8q4hl" podStartSLOduration=65.776046431 podStartE2EDuration="1m11.382928765s" totalImagesPullingTime="5.606882334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:40:58.382775835 +0000 UTC m=+72.934692254" watchObservedRunningTime="2026-07-27 19:40:58.382928765 +0000 UTC m=+72.934845191"
Jul 27 19:40:58.474548 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:58.474507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:58.477347 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:58.477319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/625ce43b-c9a8-4a13-91e3-b6e014b59c1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l46qm\" (UID: \"625ce43b-c9a8-4a13-91e3-b6e014b59c1b\") " pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:58.706144 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:58.706110    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-csj86\""
Jul 27 19:40:58.714704 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:58.714673    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l46qm"
Jul 27 19:40:58.861245 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:58.861209    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l46qm"]
Jul 27 19:40:58.864505 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:40:58.864461    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625ce43b_c9a8_4a13_91e3_b6e014b59c1b.slice/crio-8c2ce3814f09cad8b4cb476b6b7785ae385e73a2923a6440b4f333c5677ea449 WatchSource:0}: Error finding container 8c2ce3814f09cad8b4cb476b6b7785ae385e73a2923a6440b4f333c5677ea449: Status 404 returned error can't find the container with id 8c2ce3814f09cad8b4cb476b6b7785ae385e73a2923a6440b4f333c5677ea449
Jul 27 19:40:59.360349 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:40:59.360307    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l46qm" event={"ID":"625ce43b-c9a8-4a13-91e3-b6e014b59c1b","Type":"ContainerStarted","Data":"8c2ce3814f09cad8b4cb476b6b7785ae385e73a2923a6440b4f333c5677ea449"}
Jul 27 19:41:11.157608 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.157472    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xlpkj"]
Jul 27 19:41:11.182230 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.182205    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.184898 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.184880    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 19:41:11.185876 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.185860    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rvkzp\""
Jul 27 19:41:11.185939 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.185916    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 19:41:11.186311 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.186196    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 19:41:11.186311 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.186203    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 19:41:11.186598 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.186581    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 19:41:11.193060 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.190753    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 19:41:11.276702 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.276661    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-root\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.276702 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.276708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.277053 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.276789    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-sys\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.277053 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.276839    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-textfile\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.277053 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.276888    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-metrics-client-ca\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.277053 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.276943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.277053 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.276990    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnxtt\" (UniqueName: \"kubernetes.io/projected/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-kube-api-access-lnxtt\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.277053 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.277011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-tls\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.277053 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.277034    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-wtmp\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.377713 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.377676    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnxtt\" (UniqueName: \"kubernetes.io/projected/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-kube-api-access-lnxtt\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.377713 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.377721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-tls\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.377979 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.377741    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-wtmp\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.377979 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.377782    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-root\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.377979 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.377811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.377979 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.377896    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-root\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.377979 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.377942    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-sys\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.377979 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.377974    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-sys\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.378326 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.377985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-textfile\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.378326 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.377944    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-wtmp\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.378326 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.378020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-metrics-client-ca\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.378326 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.378056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.378326 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.378292    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-textfile\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.378706 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.378471    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.378706 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.378585    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-metrics-client-ca\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.380552 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.380528    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.380685 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.380666    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-node-exporter-tls\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.387406 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.387380    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnxtt\" (UniqueName: \"kubernetes.io/projected/fa8550e8-f12a-418a-85fb-5a39fe8f8f3c-kube-api-access-lnxtt\") pod \"node-exporter-xlpkj\" (UID: \"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c\") " pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:11.494368 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:11.494288    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xlpkj"
Jul 27 19:41:12.013610 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:41:12.013572    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8550e8_f12a_418a_85fb_5a39fe8f8f3c.slice/crio-b1cf6d38fccfea19a1f366b4a2e4295e1169b412e4f42680218bb07be80f49f5 WatchSource:0}: Error finding container b1cf6d38fccfea19a1f366b4a2e4295e1169b412e4f42680218bb07be80f49f5: Status 404 returned error can't find the container with id b1cf6d38fccfea19a1f366b4a2e4295e1169b412e4f42680218bb07be80f49f5
Jul 27 19:41:12.413436 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:12.413391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-w6bsx" event={"ID":"db3b9846-fa9a-40d0-b2fa-031c0c6212b2","Type":"ContainerStarted","Data":"2eafb1b351f8a2986d333bff93c307a8b8f32c35da86df9da24a779e14671676"}
Jul 27 19:41:12.413920 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:12.413633    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-w6bsx"
Jul 27 19:41:12.416763 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:12.416726    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xlpkj" event={"ID":"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c","Type":"ContainerStarted","Data":"b1cf6d38fccfea19a1f366b4a2e4295e1169b412e4f42680218bb07be80f49f5"}
Jul 27 19:41:12.419621 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:12.419593    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l46qm" event={"ID":"625ce43b-c9a8-4a13-91e3-b6e014b59c1b","Type":"ContainerStarted","Data":"41829afac4d0f879cb391ffa174d14a00110c51bcca6cc17a994b62b9555076c"}
Jul 27 19:41:12.424686 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:12.424669    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-w6bsx"
Jul 27 19:41:12.434006 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:12.433729    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-w6bsx" podStartSLOduration=1.6421701039999999 podStartE2EDuration="20.433709936s" totalImagesPullingTime="18.791539832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:12.432109756 +0000 UTC m=+86.984026178" watchObservedRunningTime="2026-07-27 19:41:12.433709936 +0000 UTC m=+86.985626355"
Jul 27 19:41:14.433317 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:14.433204    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xlpkj" event={"ID":"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c","Type":"ContainerStarted","Data":"08a1e705588a1a979d7b9ed4c7850158f760fee17b1017430e1470fc0ac95dc1"}
Jul 27 19:41:15.847440 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:15.847404    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"]
Jul 27 19:41:15.867873 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:15.867836    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"]
Jul 27 19:41:15.868043 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:15.867930    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"
Jul 27 19:41:15.870246 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:15.870076    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tj92g\""
Jul 27 19:41:15.870246 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:15.870151    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 27 19:41:16.026044 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.026008    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz2xv\" (UniqueName: \"kubernetes.io/projected/527491fe-e2e2-4855-87b7-57377502f9a5-kube-api-access-lz2xv\") pod \"monitoring-plugin-68b7fc9574-tn62g\" (UID: \"527491fe-e2e2-4855-87b7-57377502f9a5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"
Jul 27 19:41:16.026184 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.026063    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/527491fe-e2e2-4855-87b7-57377502f9a5-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tn62g\" (UID: \"527491fe-e2e2-4855-87b7-57377502f9a5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"
Jul 27 19:41:16.126598 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.126413    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz2xv\" (UniqueName: \"kubernetes.io/projected/527491fe-e2e2-4855-87b7-57377502f9a5-kube-api-access-lz2xv\") pod \"monitoring-plugin-68b7fc9574-tn62g\" (UID: \"527491fe-e2e2-4855-87b7-57377502f9a5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"
Jul 27 19:41:16.126812 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.126470    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/527491fe-e2e2-4855-87b7-57377502f9a5-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tn62g\" (UID: \"527491fe-e2e2-4855-87b7-57377502f9a5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"
Jul 27 19:41:16.127557 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:41:16.126980    2562 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 27 19:41:16.127557 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:41:16.127053    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/527491fe-e2e2-4855-87b7-57377502f9a5-monitoring-plugin-cert podName:527491fe-e2e2-4855-87b7-57377502f9a5 nodeName:}" failed. No retries permitted until 2026-07-27 19:41:16.627029748 +0000 UTC m=+91.178946146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/527491fe-e2e2-4855-87b7-57377502f9a5-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-tn62g" (UID: "527491fe-e2e2-4855-87b7-57377502f9a5") : secret "monitoring-plugin-cert" not found
Jul 27 19:41:16.136344 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.136320    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz2xv\" (UniqueName: \"kubernetes.io/projected/527491fe-e2e2-4855-87b7-57377502f9a5-kube-api-access-lz2xv\") pod \"monitoring-plugin-68b7fc9574-tn62g\" (UID: \"527491fe-e2e2-4855-87b7-57377502f9a5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"
Jul 27 19:41:16.354497 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.351392    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78f95f545c-r7f6p"
Jul 27 19:41:16.388300 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.388183    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"]
Jul 27 19:41:16.445438 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.445405    2562 generic.go:356] "Generic (PLEG): container finished" podID="fa8550e8-f12a-418a-85fb-5a39fe8f8f3c" containerID="08a1e705588a1a979d7b9ed4c7850158f760fee17b1017430e1470fc0ac95dc1" exitCode=0
Jul 27 19:41:16.470798 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.470761    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"]
Jul 27 19:41:16.470976 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.470817    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xlpkj" event={"ID":"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c","Type":"ContainerDied","Data":"08a1e705588a1a979d7b9ed4c7850158f760fee17b1017430e1470fc0ac95dc1"}
Jul 27 19:41:16.471626 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.471600    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.474898 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.474875    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-pp586\""
Jul 27 19:41:16.475054 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.475033    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 27 19:41:16.475199 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.475179    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 27 19:41:16.475284 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.475259    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 27 19:41:16.477157 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.477136    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 27 19:41:16.477791 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.477327    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 27 19:41:16.489317 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.489284    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 27 19:41:16.633351 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.633309    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a24af1c0-25af-4a5d-9329-d017e80af81b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.633555 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.633369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a24af1c0-25af-4a5d-9329-d017e80af81b-secret-telemeter-client\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.633555 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.633393    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q5bv\" (UniqueName: \"kubernetes.io/projected/a24af1c0-25af-4a5d-9329-d017e80af81b-kube-api-access-8q5bv\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.633555 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.633504    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/527491fe-e2e2-4855-87b7-57377502f9a5-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tn62g\" (UID: \"527491fe-e2e2-4855-87b7-57377502f9a5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"
Jul 27 19:41:16.633726 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.633559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a24af1c0-25af-4a5d-9329-d017e80af81b-telemeter-client-tls\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.633726 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.633606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a24af1c0-25af-4a5d-9329-d017e80af81b-federate-client-tls\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.633726 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.633632    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a24af1c0-25af-4a5d-9329-d017e80af81b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.633726 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:41:16.633663    2562 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 27 19:41:16.633914 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:41:16.633736    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/527491fe-e2e2-4855-87b7-57377502f9a5-monitoring-plugin-cert podName:527491fe-e2e2-4855-87b7-57377502f9a5 nodeName:}" failed. No retries permitted until 2026-07-27 19:41:17.633713745 +0000 UTC m=+92.185630143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/527491fe-e2e2-4855-87b7-57377502f9a5-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-tn62g" (UID: "527491fe-e2e2-4855-87b7-57377502f9a5") : secret "monitoring-plugin-cert" not found
Jul 27 19:41:16.633914 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.633773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a24af1c0-25af-4a5d-9329-d017e80af81b-metrics-client-ca\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.633914 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.633818    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a24af1c0-25af-4a5d-9329-d017e80af81b-serving-certs-ca-bundle\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.734915 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.734839    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a24af1c0-25af-4a5d-9329-d017e80af81b-telemeter-client-tls\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.734915 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.734891    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a24af1c0-25af-4a5d-9329-d017e80af81b-federate-client-tls\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.735132 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.735028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a24af1c0-25af-4a5d-9329-d017e80af81b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.735204 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.735184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a24af1c0-25af-4a5d-9329-d017e80af81b-metrics-client-ca\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.735291 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.735216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a24af1c0-25af-4a5d-9329-d017e80af81b-serving-certs-ca-bundle\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.735291 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.735252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a24af1c0-25af-4a5d-9329-d017e80af81b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.735291 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.735281    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a24af1c0-25af-4a5d-9329-d017e80af81b-secret-telemeter-client\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.735436 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.735309    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q5bv\" (UniqueName: \"kubernetes.io/projected/a24af1c0-25af-4a5d-9329-d017e80af81b-kube-api-access-8q5bv\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.736848 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.736017    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a24af1c0-25af-4a5d-9329-d017e80af81b-serving-certs-ca-bundle\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.736848 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.736072    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a24af1c0-25af-4a5d-9329-d017e80af81b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.736848 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.736651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a24af1c0-25af-4a5d-9329-d017e80af81b-metrics-client-ca\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.739290 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.739113    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a24af1c0-25af-4a5d-9329-d017e80af81b-secret-telemeter-client\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.739981 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.739938    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a24af1c0-25af-4a5d-9329-d017e80af81b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.740945 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.740919    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a24af1c0-25af-4a5d-9329-d017e80af81b-telemeter-client-tls\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.751030 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.750962    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a24af1c0-25af-4a5d-9329-d017e80af81b-federate-client-tls\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.754041 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.754014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q5bv\" (UniqueName: \"kubernetes.io/projected/a24af1c0-25af-4a5d-9329-d017e80af81b-kube-api-access-8q5bv\") pod \"telemeter-client-7f998b6b8f-jqfgd\" (UID: \"a24af1c0-25af-4a5d-9329-d017e80af81b\") " pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:16.789573 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:16.789536    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"
Jul 27 19:41:17.228596 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.228552    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59998db7cd-65pzp"]
Jul 27 19:41:17.236802 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:41:17.236770    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24af1c0_25af_4a5d_9329_d017e80af81b.slice/crio-5a04fc29425383c6ddeb564b82d2733b8621e61e53a0f5212043dd36b72f5e78 WatchSource:0}: Error finding container 5a04fc29425383c6ddeb564b82d2733b8621e61e53a0f5212043dd36b72f5e78: Status 404 returned error can't find the container with id 5a04fc29425383c6ddeb564b82d2733b8621e61e53a0f5212043dd36b72f5e78
Jul 27 19:41:17.251998 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.251974    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd"]
Jul 27 19:41:17.252118 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.252025    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59998db7cd-65pzp"]
Jul 27 19:41:17.252118 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.252089    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.254248 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.254227    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sj6nc\""
Jul 27 19:41:17.254631 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.254613    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 19:41:17.255956 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.254860    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 19:41:17.255956 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.255054    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 19:41:17.255956 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.255306    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 19:41:17.255956 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.255520    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 19:41:17.260179 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.260161    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 19:41:17.416208 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.416180    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:41:17.437645 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.437611    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:41:17.437787 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.437773    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.440337 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.440313    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 19:41:17.440908 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.440519    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eg6i20k34ans7\""
Jul 27 19:41:17.440908 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.440556    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 19:41:17.440908 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.440420    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-oauth-serving-cert\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.440908 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.440731    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-trusted-ca-bundle\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.441167 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.440952    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-service-ca\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.441328 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.441270    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be68d266-a08f-475f-ba80-1ec512a2edcd-console-serving-cert\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.441430 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.441385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be68d266-a08f-475f-ba80-1ec512a2edcd-console-oauth-config\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.441430 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.441407    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 19:41:17.441430 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.441412    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 19:41:17.441623 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.441457    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 19:41:17.441623 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.441604    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-console-config\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.441720 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.441669    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cbbwn\""
Jul 27 19:41:17.441760 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.441740    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 19:41:17.441834 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.441800    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 19:41:17.442444 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.441952    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjt7h\" (UniqueName: \"kubernetes.io/projected/be68d266-a08f-475f-ba80-1ec512a2edcd-kube-api-access-vjt7h\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.442444 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.441978    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 19:41:17.442444 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.442014    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 19:41:17.442444 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.442118    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 19:41:17.442444 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.442125    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 19:41:17.442444 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.442244    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 19:41:17.442444 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.442246    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 19:41:17.444877 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.444859    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 19:41:17.448945 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.448918    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 19:41:17.451545 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.451514    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd" event={"ID":"a24af1c0-25af-4a5d-9329-d017e80af81b","Type":"ContainerStarted","Data":"5a04fc29425383c6ddeb564b82d2733b8621e61e53a0f5212043dd36b72f5e78"}
Jul 27 19:41:17.453946 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.453924    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xlpkj" event={"ID":"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c","Type":"ContainerStarted","Data":"7cbb9980b84f4f4f7225344da208306f0efe7b0f02fda1e52f2f1990a057a21f"}
Jul 27 19:41:17.454035 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.453970    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xlpkj" event={"ID":"fa8550e8-f12a-418a-85fb-5a39fe8f8f3c","Type":"ContainerStarted","Data":"b3ee193dc9aae459f8e6ec4fa2a635dda4ade9dc1d1467bf2ca32f8ed67c617f"}
Jul 27 19:41:17.488390 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.488341    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xlpkj" podStartSLOduration=4.435200209 podStartE2EDuration="6.488323168s" totalImagesPullingTime="1.910637101s" totalInitContainerRuntime="142.485858ms" isStatefulPod=true podCreationTimestamp="2026-07-27 19:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:17.487618827 +0000 UTC m=+92.039535256" watchObservedRunningTime="2026-07-27 19:41:17.488323168 +0000 UTC m=+92.040239585"
Jul 27 19:41:17.542730 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.542702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.542896 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.542745    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.542896 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.542837    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be68d266-a08f-475f-ba80-1ec512a2edcd-console-oauth-config\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.542896 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.542874    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhx6r\" (UniqueName: \"kubernetes.io/projected/d53ab6e3-7565-40f9-bb2b-6194702d94e3-kube-api-access-bhx6r\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543072 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.542979    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-console-config\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.543072 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543049    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d53ab6e3-7565-40f9-bb2b-6194702d94e3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543181 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543094    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543181 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543120    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543181 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543330 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543233    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543330 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543266    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543330 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjt7h\" (UniqueName: \"kubernetes.io/projected/be68d266-a08f-475f-ba80-1ec512a2edcd-kube-api-access-vjt7h\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.543494 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543494 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d53ab6e3-7565-40f9-bb2b-6194702d94e3-config-out\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543494 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543395    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543494 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543434    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543494 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543461    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543737 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-oauth-serving-cert\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.543737 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-trusted-ca-bundle\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.543737 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543737 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543614    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-web-config\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543737 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543644    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-service-ca\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.543737 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.543737 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543729    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be68d266-a08f-475f-ba80-1ec512a2edcd-console-serving-cert\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.544189 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543757    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.544189 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543795    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.544189 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543856    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-config\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.544189 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.543897    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-console-config\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.545231 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.545199    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-oauth-serving-cert\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.545454 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.545438    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-service-ca\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.553038 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.546779    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be68d266-a08f-475f-ba80-1ec512a2edcd-console-oauth-config\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.553038 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.550394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-trusted-ca-bundle\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.553038 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.551985    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be68d266-a08f-475f-ba80-1ec512a2edcd-console-serving-cert\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.553038 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.552613    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjt7h\" (UniqueName: \"kubernetes.io/projected/be68d266-a08f-475f-ba80-1ec512a2edcd-kube-api-access-vjt7h\") pod \"console-59998db7cd-65pzp\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") " pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.584402 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.584216    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:17.644981 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.644945    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645147 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.644987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645147 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645147 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645093    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-web-config\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645147 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645147    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645369 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645177    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/527491fe-e2e2-4855-87b7-57377502f9a5-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tn62g\" (UID: \"527491fe-e2e2-4855-87b7-57377502f9a5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"
Jul 27 19:41:17.645369 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645369 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645232    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645369 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645261    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-config\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645369 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645369 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645340    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645670 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhx6r\" (UniqueName: \"kubernetes.io/projected/d53ab6e3-7565-40f9-bb2b-6194702d94e3-kube-api-access-bhx6r\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645670 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645413    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d53ab6e3-7565-40f9-bb2b-6194702d94e3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645670 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645670 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645670 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645670 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645670 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645670 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645623    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645670 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d53ab6e3-7565-40f9-bb2b-6194702d94e3-config-out\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.645670 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.645666    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.646978 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.646952    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.647111 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.646976    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.648469 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.648441    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.648687 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.648651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.648779 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.648694    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.649528 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.648915    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.651324 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.651278    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.653173 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.653148    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/527491fe-e2e2-4855-87b7-57377502f9a5-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tn62g\" (UID: \"527491fe-e2e2-4855-87b7-57377502f9a5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"
Jul 27 19:41:17.653268 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.653171    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.653357 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.653339    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.653446 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.653397    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.654429 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.654404    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.654685 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.654666    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d53ab6e3-7565-40f9-bb2b-6194702d94e3-config-out\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.655044 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.655015    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.655354 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.655332    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-config\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.655354 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.655350    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.655796 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.655774    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.659725 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.656213    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.659725 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.656968    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-web-config\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.659725 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.658227    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d53ab6e3-7565-40f9-bb2b-6194702d94e3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.659725 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.659448    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhx6r\" (UniqueName: \"kubernetes.io/projected/d53ab6e3-7565-40f9-bb2b-6194702d94e3-kube-api-access-bhx6r\") pod \"prometheus-k8s-0\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.680249 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.680180    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"
Jul 27 19:41:17.751401 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.751342    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:17.752839 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.752810    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59998db7cd-65pzp"]
Jul 27 19:41:17.756946 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:41:17.756902    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe68d266_a08f_475f_ba80_1ec512a2edcd.slice/crio-90c583d8a29858744028e7ba2e3a8a72a3fcd4770653733e06d343d511e53f01 WatchSource:0}: Error finding container 90c583d8a29858744028e7ba2e3a8a72a3fcd4770653733e06d343d511e53f01: Status 404 returned error can't find the container with id 90c583d8a29858744028e7ba2e3a8a72a3fcd4770653733e06d343d511e53f01
Jul 27 19:41:17.884785 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.878776    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"]
Jul 27 19:41:17.891808 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:41:17.891762    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527491fe_e2e2_4855_87b7_57377502f9a5.slice/crio-f769fa28492698665fa04299c896c40b40f267d35b72c584456c712597711b80 WatchSource:0}: Error finding container f769fa28492698665fa04299c896c40b40f267d35b72c584456c712597711b80: Status 404 returned error can't find the container with id f769fa28492698665fa04299c896c40b40f267d35b72c584456c712597711b80
Jul 27 19:41:17.958306 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:17.958161    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:41:17.961517 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:41:17.961459    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53ab6e3_7565_40f9_bb2b_6194702d94e3.slice/crio-badfc957df50d73f9ee58d933a23b2443e13ac12fa02e94f22201bfb56072f15 WatchSource:0}: Error finding container badfc957df50d73f9ee58d933a23b2443e13ac12fa02e94f22201bfb56072f15: Status 404 returned error can't find the container with id badfc957df50d73f9ee58d933a23b2443e13ac12fa02e94f22201bfb56072f15
Jul 27 19:41:18.465605 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:18.465514    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59998db7cd-65pzp" event={"ID":"be68d266-a08f-475f-ba80-1ec512a2edcd","Type":"ContainerStarted","Data":"90c583d8a29858744028e7ba2e3a8a72a3fcd4770653733e06d343d511e53f01"}
Jul 27 19:41:18.469689 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:18.469584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerStarted","Data":"badfc957df50d73f9ee58d933a23b2443e13ac12fa02e94f22201bfb56072f15"}
Jul 27 19:41:18.475278 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:18.475170    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l46qm" event={"ID":"625ce43b-c9a8-4a13-91e3-b6e014b59c1b","Type":"ContainerStarted","Data":"6743bb3329ec2373e71a34734c894f68005234f791562e9a3c1890ed8d304370"}
Jul 27 19:41:18.476815 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:18.476754    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g" event={"ID":"527491fe-e2e2-4855-87b7-57377502f9a5","Type":"ContainerStarted","Data":"f769fa28492698665fa04299c896c40b40f267d35b72c584456c712597711b80"}
Jul 27 19:41:26.520712 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:26.520673    2562 generic.go:356] "Generic (PLEG): container finished" podID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerID="ee24fc90f9cb1b5a434569c3cceb1fe47fe90862a074bcd7b1174e8b4907ea05" exitCode=0
Jul 27 19:41:26.521209 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:26.520763    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerDied","Data":"ee24fc90f9cb1b5a434569c3cceb1fe47fe90862a074bcd7b1174e8b4907ea05"}
Jul 27 19:41:26.522990 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:26.522961    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l46qm" event={"ID":"625ce43b-c9a8-4a13-91e3-b6e014b59c1b","Type":"ContainerStarted","Data":"6dc28cb118c052310af69263f32dc07c305da9c70fd522d37f6bbd7dd6fc915e"}
Jul 27 19:41:26.524784 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:26.524755    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g" event={"ID":"527491fe-e2e2-4855-87b7-57377502f9a5","Type":"ContainerStarted","Data":"ed6859563a17af538ac876e9a0f286b4b6c3cef8dee85699c97d9e8ff2c42a57"}
Jul 27 19:41:26.524917 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:26.524878    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"
Jul 27 19:41:26.527007 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:26.526972    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd" event={"ID":"a24af1c0-25af-4a5d-9329-d017e80af81b","Type":"ContainerStarted","Data":"d8c616777555c76d184d011daadc9c364794cb10cfc586480d5f0ace2f131698"}
Jul 27 19:41:26.527095 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:26.527027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd" event={"ID":"a24af1c0-25af-4a5d-9329-d017e80af81b","Type":"ContainerStarted","Data":"7bbe871bb1e75dc57974e63caeb241b1abcb83ef455122ad1c8ecedb1da6590d"}
Jul 27 19:41:26.527095 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:26.527052    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd" event={"ID":"a24af1c0-25af-4a5d-9329-d017e80af81b","Type":"ContainerStarted","Data":"181cc8edac76c638b08b3bc11197c7bfa267fe75249c315ddb67101d15d7c4ac"}
Jul 27 19:41:26.528972 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:26.528950    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59998db7cd-65pzp" event={"ID":"be68d266-a08f-475f-ba80-1ec512a2edcd","Type":"ContainerStarted","Data":"bb6eb4b98da13bc73a8c03996facc308e4dfc4fbee497e3e66c72cbe02756546"}
Jul 27 19:41:26.530061 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:26.530043    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g"
Jul 27 19:41:26.567961 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:26.567917    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l46qm" podStartSLOduration=33.874844359 podStartE2EDuration="1m0.567899873s" totalImagesPullingTime="26.693055514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:40:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:26.567088703 +0000 UTC m=+101.119005122" watchObservedRunningTime="2026-07-27 19:41:26.567899873 +0000 UTC m=+101.119816292"
Jul 27 19:41:26.584826 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:26.584778    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tn62g" podStartSLOduration=4.027219642 podStartE2EDuration="11.584763316s" totalImagesPullingTime="7.557543674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:26.584637626 +0000 UTC m=+101.136554044" watchObservedRunningTime="2026-07-27 19:41:26.584763316 +0000 UTC m=+101.136679727"
Jul 27 19:41:26.603921 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:26.603872    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59998db7cd-65pzp" podStartSLOduration=1.903443141 podStartE2EDuration="9.603853901s" totalImagesPullingTime="7.70041076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:26.603130081 +0000 UTC m=+101.155046499" watchObservedRunningTime="2026-07-27 19:41:26.603853901 +0000 UTC m=+101.155770322"
Jul 27 19:41:26.625783 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:26.625736    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f998b6b8f-jqfgd" podStartSLOduration=2.411536651 podStartE2EDuration="10.625721802s" totalImagesPullingTime="8.214185151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:26.624896553 +0000 UTC m=+101.176812983" watchObservedRunningTime="2026-07-27 19:41:26.625721802 +0000 UTC m=+101.177638220"
Jul 27 19:41:27.585029 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:27.584992    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:27.585536 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:27.585060    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:27.590931 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:27.590904    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:28.244689 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:28.244648    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59998db7cd-65pzp"]
Jul 27 19:41:28.541820 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:28.541743    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:29.363697 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:29.363659    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8q4hl"
Jul 27 19:41:30.547517 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:30.547420    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerStarted","Data":"4c2b8b1be5bc4e4a8264fcc71da2570d9e237a2df57428648c4669e2473ae248"}
Jul 27 19:41:30.547517 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:30.547499    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerStarted","Data":"30c504b419393708502b2e56734d3003f7d0b7b0da40e808030053d25de0678c"}
Jul 27 19:41:32.557468 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:32.557439    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerStarted","Data":"ca86e1b786959806db89b7f69a01ab11d2ad342f470acf41ca36b3048f6dc854"}
Jul 27 19:41:33.565016 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:33.564982    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerStarted","Data":"e1e975574a9f3264e0d45aced12c8cab7b0f62c8da375f4cd72114d93e7fd106"}
Jul 27 19:41:33.565616 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:33.565058    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerStarted","Data":"989b625900679b834b6c2cfe84a674aaae59846899e1c6a062cf70fb89f0e3c6"}
Jul 27 19:41:33.565616 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:33.565100    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerStarted","Data":"6142bd7ece87c8e7de056dc2ddd2cbf6b27a507d0b8ddf3952e49ebe073cae47"}
Jul 27 19:41:33.596650 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:33.596595    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.47839834 podStartE2EDuration="16.596573117s" totalImagesPullingTime="13.098976722s" totalInitContainerRuntime="19.198055ms" isStatefulPod=false podCreationTimestamp="2026-07-27 19:41:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:33.595229546 +0000 UTC m=+108.147145965" watchObservedRunningTime="2026-07-27 19:41:33.596573117 +0000 UTC m=+108.148489535"
Jul 27 19:41:37.752090 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:37.752047    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:41:54.560577 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:54.560531    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59998db7cd-65pzp" podUID="be68d266-a08f-475f-ba80-1ec512a2edcd" containerName="console" containerID="cri-o://bb6eb4b98da13bc73a8c03996facc308e4dfc4fbee497e3e66c72cbe02756546" gracePeriod=15
Jul 27 19:41:55.339044 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.339009    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:55.469099 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.469009    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-oauth-serving-cert\") pod \"be68d266-a08f-475f-ba80-1ec512a2edcd\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") "
Jul 27 19:41:55.469099 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.469063    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be68d266-a08f-475f-ba80-1ec512a2edcd-console-oauth-config\") pod \"be68d266-a08f-475f-ba80-1ec512a2edcd\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") "
Jul 27 19:41:55.469099 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.469094    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-service-ca\") pod \"be68d266-a08f-475f-ba80-1ec512a2edcd\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") "
Jul 27 19:41:55.469336 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.469122    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-trusted-ca-bundle\") pod \"be68d266-a08f-475f-ba80-1ec512a2edcd\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") "
Jul 27 19:41:55.469336 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.469167    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be68d266-a08f-475f-ba80-1ec512a2edcd-console-serving-cert\") pod \"be68d266-a08f-475f-ba80-1ec512a2edcd\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") "
Jul 27 19:41:55.469336 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.469213    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-console-config\") pod \"be68d266-a08f-475f-ba80-1ec512a2edcd\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") "
Jul 27 19:41:55.469336 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.469246    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjt7h\" (UniqueName: \"kubernetes.io/projected/be68d266-a08f-475f-ba80-1ec512a2edcd-kube-api-access-vjt7h\") pod \"be68d266-a08f-475f-ba80-1ec512a2edcd\" (UID: \"be68d266-a08f-475f-ba80-1ec512a2edcd\") "
Jul 27 19:41:55.469625 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.469594    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-service-ca" (OuterVolumeSpecName: "service-ca") pod "be68d266-a08f-475f-ba80-1ec512a2edcd" (UID: "be68d266-a08f-475f-ba80-1ec512a2edcd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:41:55.469707 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.469501    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "be68d266-a08f-475f-ba80-1ec512a2edcd" (UID: "be68d266-a08f-475f-ba80-1ec512a2edcd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:41:55.469806 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.469780    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-console-config" (OuterVolumeSpecName: "console-config") pod "be68d266-a08f-475f-ba80-1ec512a2edcd" (UID: "be68d266-a08f-475f-ba80-1ec512a2edcd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:41:55.469857 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.469820    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "be68d266-a08f-475f-ba80-1ec512a2edcd" (UID: "be68d266-a08f-475f-ba80-1ec512a2edcd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:41:55.471594 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.471572    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be68d266-a08f-475f-ba80-1ec512a2edcd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "be68d266-a08f-475f-ba80-1ec512a2edcd" (UID: "be68d266-a08f-475f-ba80-1ec512a2edcd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:41:55.472007 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.471975    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be68d266-a08f-475f-ba80-1ec512a2edcd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "be68d266-a08f-475f-ba80-1ec512a2edcd" (UID: "be68d266-a08f-475f-ba80-1ec512a2edcd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:41:55.472071 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.472002    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be68d266-a08f-475f-ba80-1ec512a2edcd-kube-api-access-vjt7h" (OuterVolumeSpecName: "kube-api-access-vjt7h") pod "be68d266-a08f-475f-ba80-1ec512a2edcd" (UID: "be68d266-a08f-475f-ba80-1ec512a2edcd"). InnerVolumeSpecName "kube-api-access-vjt7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:41:55.570632 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.570596    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be68d266-a08f-475f-ba80-1ec512a2edcd-console-serving-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:41:55.570632 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.570626    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-console-config\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:41:55.570632 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.570635    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjt7h\" (UniqueName: \"kubernetes.io/projected/be68d266-a08f-475f-ba80-1ec512a2edcd-kube-api-access-vjt7h\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:41:55.570632 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.570644    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-oauth-serving-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:41:55.571085 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.570655    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be68d266-a08f-475f-ba80-1ec512a2edcd-console-oauth-config\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:41:55.571085 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.570663    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:41:55.571085 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.570671    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68d266-a08f-475f-ba80-1ec512a2edcd-trusted-ca-bundle\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:41:55.642370 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.642326    2562 generic.go:356] "Generic (PLEG): container finished" podID="be68d266-a08f-475f-ba80-1ec512a2edcd" containerID="bb6eb4b98da13bc73a8c03996facc308e4dfc4fbee497e3e66c72cbe02756546" exitCode=0
Jul 27 19:41:55.642526 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.642382    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59998db7cd-65pzp"
Jul 27 19:41:55.642526 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.642399    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59998db7cd-65pzp" event={"ID":"be68d266-a08f-475f-ba80-1ec512a2edcd","Type":"ContainerDied","Data":"bb6eb4b98da13bc73a8c03996facc308e4dfc4fbee497e3e66c72cbe02756546"}
Jul 27 19:41:55.642526 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.642442    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59998db7cd-65pzp" event={"ID":"be68d266-a08f-475f-ba80-1ec512a2edcd","Type":"ContainerDied","Data":"90c583d8a29858744028e7ba2e3a8a72a3fcd4770653733e06d343d511e53f01"}
Jul 27 19:41:55.642526 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.642463    2562 scope.go:117] "RemoveContainer" containerID="bb6eb4b98da13bc73a8c03996facc308e4dfc4fbee497e3e66c72cbe02756546"
Jul 27 19:41:55.651371 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.651351    2562 scope.go:117] "RemoveContainer" containerID="bb6eb4b98da13bc73a8c03996facc308e4dfc4fbee497e3e66c72cbe02756546"
Jul 27 19:41:55.651735 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:41:55.651715    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6eb4b98da13bc73a8c03996facc308e4dfc4fbee497e3e66c72cbe02756546\": container with ID starting with bb6eb4b98da13bc73a8c03996facc308e4dfc4fbee497e3e66c72cbe02756546 not found: ID does not exist" containerID="bb6eb4b98da13bc73a8c03996facc308e4dfc4fbee497e3e66c72cbe02756546"
Jul 27 19:41:55.651791 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.651745    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6eb4b98da13bc73a8c03996facc308e4dfc4fbee497e3e66c72cbe02756546"} err="failed to get container status \"bb6eb4b98da13bc73a8c03996facc308e4dfc4fbee497e3e66c72cbe02756546\": rpc error: code = NotFound desc = could not find container \"bb6eb4b98da13bc73a8c03996facc308e4dfc4fbee497e3e66c72cbe02756546\": container with ID starting with bb6eb4b98da13bc73a8c03996facc308e4dfc4fbee497e3e66c72cbe02756546 not found: ID does not exist"
Jul 27 19:41:55.660990 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.660961    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59998db7cd-65pzp"]
Jul 27 19:41:55.665136 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:55.665115    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59998db7cd-65pzp"]
Jul 27 19:41:56.009432 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:41:56.009391    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be68d266-a08f-475f-ba80-1ec512a2edcd" path="/var/lib/kubelet/pods/be68d266-a08f-475f-ba80-1ec512a2edcd/volumes"
Jul 27 19:42:17.751885 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:17.751848    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:17.771001 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:17.770976    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:18.744200 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:18.744163    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:35.820207 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:35.820172    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:42:35.820716 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:35.820468    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="prometheus" containerID="cri-o://30c504b419393708502b2e56734d3003f7d0b7b0da40e808030053d25de0678c" gracePeriod=600
Jul 27 19:42:35.820716 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:35.820500    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="kube-rbac-proxy" containerID="cri-o://989b625900679b834b6c2cfe84a674aaae59846899e1c6a062cf70fb89f0e3c6" gracePeriod=600
Jul 27 19:42:35.820716 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:35.820558    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e1e975574a9f3264e0d45aced12c8cab7b0f62c8da375f4cd72114d93e7fd106" gracePeriod=600
Jul 27 19:42:35.820716 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:35.820606    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="config-reloader" containerID="cri-o://4c2b8b1be5bc4e4a8264fcc71da2570d9e237a2df57428648c4669e2473ae248" gracePeriod=600
Jul 27 19:42:35.820716 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:35.820558    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="thanos-sidecar" containerID="cri-o://ca86e1b786959806db89b7f69a01ab11d2ad342f470acf41ca36b3048f6dc854" gracePeriod=600
Jul 27 19:42:35.820944 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:35.820561    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="kube-rbac-proxy-web" containerID="cri-o://6142bd7ece87c8e7de056dc2ddd2cbf6b27a507d0b8ddf3952e49ebe073cae47" gracePeriod=600
Jul 27 19:42:36.792029 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:36.791944    2562 generic.go:356] "Generic (PLEG): container finished" podID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerID="e1e975574a9f3264e0d45aced12c8cab7b0f62c8da375f4cd72114d93e7fd106" exitCode=0
Jul 27 19:42:36.792029 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:36.791967    2562 generic.go:356] "Generic (PLEG): container finished" podID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerID="989b625900679b834b6c2cfe84a674aaae59846899e1c6a062cf70fb89f0e3c6" exitCode=0
Jul 27 19:42:36.792029 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:36.791975    2562 generic.go:356] "Generic (PLEG): container finished" podID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerID="ca86e1b786959806db89b7f69a01ab11d2ad342f470acf41ca36b3048f6dc854" exitCode=0
Jul 27 19:42:36.792029 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:36.791984    2562 generic.go:356] "Generic (PLEG): container finished" podID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerID="4c2b8b1be5bc4e4a8264fcc71da2570d9e237a2df57428648c4669e2473ae248" exitCode=0
Jul 27 19:42:36.792029 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:36.791989    2562 generic.go:356] "Generic (PLEG): container finished" podID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerID="30c504b419393708502b2e56734d3003f7d0b7b0da40e808030053d25de0678c" exitCode=0
Jul 27 19:42:36.792029 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:36.792012    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerDied","Data":"e1e975574a9f3264e0d45aced12c8cab7b0f62c8da375f4cd72114d93e7fd106"}
Jul 27 19:42:36.792342 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:36.792075    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerDied","Data":"989b625900679b834b6c2cfe84a674aaae59846899e1c6a062cf70fb89f0e3c6"}
Jul 27 19:42:36.792342 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:36.792102    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerDied","Data":"ca86e1b786959806db89b7f69a01ab11d2ad342f470acf41ca36b3048f6dc854"}
Jul 27 19:42:36.792342 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:36.792125    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerDied","Data":"4c2b8b1be5bc4e4a8264fcc71da2570d9e237a2df57428648c4669e2473ae248"}
Jul 27 19:42:36.792342 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:36.792149    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerDied","Data":"30c504b419393708502b2e56734d3003f7d0b7b0da40e808030053d25de0678c"}
Jul 27 19:42:37.087638 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.087614    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.214757 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.214717    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-thanos-prometheus-http-client-file\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.214757 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.214758    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d53ab6e3-7565-40f9-bb2b-6194702d94e3-config-out\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215004 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.214855    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-tls\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215004 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.214898    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-metrics-client-certs\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215004 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.214919    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-kubelet-serving-ca-bundle\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215004 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.214939    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-serving-certs-ca-bundle\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215004 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.214959    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-0\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215004 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.214991    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-grpc-tls\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215299 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215028    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215299 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215052    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-config\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215299 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215078    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-db\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215299 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215113    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhx6r\" (UniqueName: \"kubernetes.io/projected/d53ab6e3-7565-40f9-bb2b-6194702d94e3-kube-api-access-bhx6r\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215299 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215135    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-web-config\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215299 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215174    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-1\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215299 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215200    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d53ab6e3-7565-40f9-bb2b-6194702d94e3-tls-assets\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215299 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215230    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-2\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215299 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215261    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215299 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215288    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-trusted-ca-bundle\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215974 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215333    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-metrics-client-ca\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215974 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215358    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:42:37.215974 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215372    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-kube-rbac-proxy\") pod \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\" (UID: \"d53ab6e3-7565-40f9-bb2b-6194702d94e3\") "
Jul 27 19:42:37.215974 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215610    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.215974 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215617    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:42:37.215974 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.215898    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:42:37.218197 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.218168    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:42:37.218332 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.218227    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:42:37.218332 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.218285    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:42:37.218764 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.218740    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:42:37.218868 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.218853    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:42:37.219102 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.218863    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:42:37.219102 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.218928    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53ab6e3-7565-40f9-bb2b-6194702d94e3-config-out" (OuterVolumeSpecName: "config-out") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:42:37.219102 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.218993    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:42:37.219553 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.219529    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:42:37.219827 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.219801    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:42:37.219898 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.219819    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:42:37.219959 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.219921    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:42:37.220196 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.220171    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:42:37.220752 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.220733    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53ab6e3-7565-40f9-bb2b-6194702d94e3-kube-api-access-bhx6r" (OuterVolumeSpecName: "kube-api-access-bhx6r") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "kube-api-access-bhx6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:42:37.220963 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.220934    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53ab6e3-7565-40f9-bb2b-6194702d94e3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:42:37.221038 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.221002    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-config" (OuterVolumeSpecName: "config") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:42:37.231635 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.231609    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-web-config" (OuterVolumeSpecName: "web-config") pod "d53ab6e3-7565-40f9-bb2b-6194702d94e3" (UID: "d53ab6e3-7565-40f9-bb2b-6194702d94e3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:42:37.315955 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.315927    2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d53ab6e3-7565-40f9-bb2b-6194702d94e3-config-out\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.315955 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.315951    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.315955 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.315963    2562 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-metrics-client-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.315972    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.315981    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.315989    2562 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-grpc-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.316000    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.316009    2562 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-config\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.316017    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-db\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.316026    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhx6r\" (UniqueName: \"kubernetes.io/projected/d53ab6e3-7565-40f9-bb2b-6194702d94e3-kube-api-access-bhx6r\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.316034    2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-web-config\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.316041    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.316050    2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d53ab6e3-7565-40f9-bb2b-6194702d94e3-tls-assets\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.316059    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.316067    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.316075    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.316084    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d53ab6e3-7565-40f9-bb2b-6194702d94e3-configmap-metrics-client-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.316092    2562 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-secret-kube-rbac-proxy\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.316175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.316100    2562 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d53ab6e3-7565-40f9-bb2b-6194702d94e3-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:42:37.802226 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.802187    2562 generic.go:356] "Generic (PLEG): container finished" podID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerID="6142bd7ece87c8e7de056dc2ddd2cbf6b27a507d0b8ddf3952e49ebe073cae47" exitCode=0
Jul 27 19:42:37.802368 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.802260    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerDied","Data":"6142bd7ece87c8e7de056dc2ddd2cbf6b27a507d0b8ddf3952e49ebe073cae47"}
Jul 27 19:42:37.802368 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.802287    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.802368 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.802349    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d53ab6e3-7565-40f9-bb2b-6194702d94e3","Type":"ContainerDied","Data":"badfc957df50d73f9ee58d933a23b2443e13ac12fa02e94f22201bfb56072f15"}
Jul 27 19:42:37.802494 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.802382    2562 scope.go:117] "RemoveContainer" containerID="e1e975574a9f3264e0d45aced12c8cab7b0f62c8da375f4cd72114d93e7fd106"
Jul 27 19:42:37.810581 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.810549    2562 scope.go:117] "RemoveContainer" containerID="989b625900679b834b6c2cfe84a674aaae59846899e1c6a062cf70fb89f0e3c6"
Jul 27 19:42:37.817711 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.817691    2562 scope.go:117] "RemoveContainer" containerID="6142bd7ece87c8e7de056dc2ddd2cbf6b27a507d0b8ddf3952e49ebe073cae47"
Jul 27 19:42:37.824854 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.824835    2562 scope.go:117] "RemoveContainer" containerID="ca86e1b786959806db89b7f69a01ab11d2ad342f470acf41ca36b3048f6dc854"
Jul 27 19:42:37.826363 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.826345    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:42:37.832330 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.832312    2562 scope.go:117] "RemoveContainer" containerID="4c2b8b1be5bc4e4a8264fcc71da2570d9e237a2df57428648c4669e2473ae248"
Jul 27 19:42:37.833587 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.833568    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:42:37.839078 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.839062    2562 scope.go:117] "RemoveContainer" containerID="30c504b419393708502b2e56734d3003f7d0b7b0da40e808030053d25de0678c"
Jul 27 19:42:37.846162 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.846145    2562 scope.go:117] "RemoveContainer" containerID="ee24fc90f9cb1b5a434569c3cceb1fe47fe90862a074bcd7b1174e8b4907ea05"
Jul 27 19:42:37.854657 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.854641    2562 scope.go:117] "RemoveContainer" containerID="e1e975574a9f3264e0d45aced12c8cab7b0f62c8da375f4cd72114d93e7fd106"
Jul 27 19:42:37.854912 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:42:37.854891    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e975574a9f3264e0d45aced12c8cab7b0f62c8da375f4cd72114d93e7fd106\": container with ID starting with e1e975574a9f3264e0d45aced12c8cab7b0f62c8da375f4cd72114d93e7fd106 not found: ID does not exist" containerID="e1e975574a9f3264e0d45aced12c8cab7b0f62c8da375f4cd72114d93e7fd106"
Jul 27 19:42:37.854981 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.854920    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e975574a9f3264e0d45aced12c8cab7b0f62c8da375f4cd72114d93e7fd106"} err="failed to get container status \"e1e975574a9f3264e0d45aced12c8cab7b0f62c8da375f4cd72114d93e7fd106\": rpc error: code = NotFound desc = could not find container \"e1e975574a9f3264e0d45aced12c8cab7b0f62c8da375f4cd72114d93e7fd106\": container with ID starting with e1e975574a9f3264e0d45aced12c8cab7b0f62c8da375f4cd72114d93e7fd106 not found: ID does not exist"
Jul 27 19:42:37.854981 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.854940    2562 scope.go:117] "RemoveContainer" containerID="989b625900679b834b6c2cfe84a674aaae59846899e1c6a062cf70fb89f0e3c6"
Jul 27 19:42:37.855196 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:42:37.855179    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989b625900679b834b6c2cfe84a674aaae59846899e1c6a062cf70fb89f0e3c6\": container with ID starting with 989b625900679b834b6c2cfe84a674aaae59846899e1c6a062cf70fb89f0e3c6 not found: ID does not exist" containerID="989b625900679b834b6c2cfe84a674aaae59846899e1c6a062cf70fb89f0e3c6"
Jul 27 19:42:37.855235 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.855202    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989b625900679b834b6c2cfe84a674aaae59846899e1c6a062cf70fb89f0e3c6"} err="failed to get container status \"989b625900679b834b6c2cfe84a674aaae59846899e1c6a062cf70fb89f0e3c6\": rpc error: code = NotFound desc = could not find container \"989b625900679b834b6c2cfe84a674aaae59846899e1c6a062cf70fb89f0e3c6\": container with ID starting with 989b625900679b834b6c2cfe84a674aaae59846899e1c6a062cf70fb89f0e3c6 not found: ID does not exist"
Jul 27 19:42:37.855235 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.855218    2562 scope.go:117] "RemoveContainer" containerID="6142bd7ece87c8e7de056dc2ddd2cbf6b27a507d0b8ddf3952e49ebe073cae47"
Jul 27 19:42:37.855458 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:42:37.855444    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6142bd7ece87c8e7de056dc2ddd2cbf6b27a507d0b8ddf3952e49ebe073cae47\": container with ID starting with 6142bd7ece87c8e7de056dc2ddd2cbf6b27a507d0b8ddf3952e49ebe073cae47 not found: ID does not exist" containerID="6142bd7ece87c8e7de056dc2ddd2cbf6b27a507d0b8ddf3952e49ebe073cae47"
Jul 27 19:42:37.855533 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.855461    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6142bd7ece87c8e7de056dc2ddd2cbf6b27a507d0b8ddf3952e49ebe073cae47"} err="failed to get container status \"6142bd7ece87c8e7de056dc2ddd2cbf6b27a507d0b8ddf3952e49ebe073cae47\": rpc error: code = NotFound desc = could not find container \"6142bd7ece87c8e7de056dc2ddd2cbf6b27a507d0b8ddf3952e49ebe073cae47\": container with ID starting with 6142bd7ece87c8e7de056dc2ddd2cbf6b27a507d0b8ddf3952e49ebe073cae47 not found: ID does not exist"
Jul 27 19:42:37.855533 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.855489    2562 scope.go:117] "RemoveContainer" containerID="ca86e1b786959806db89b7f69a01ab11d2ad342f470acf41ca36b3048f6dc854"
Jul 27 19:42:37.855685 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:42:37.855673    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca86e1b786959806db89b7f69a01ab11d2ad342f470acf41ca36b3048f6dc854\": container with ID starting with ca86e1b786959806db89b7f69a01ab11d2ad342f470acf41ca36b3048f6dc854 not found: ID does not exist" containerID="ca86e1b786959806db89b7f69a01ab11d2ad342f470acf41ca36b3048f6dc854"
Jul 27 19:42:37.855722 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.855687    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca86e1b786959806db89b7f69a01ab11d2ad342f470acf41ca36b3048f6dc854"} err="failed to get container status \"ca86e1b786959806db89b7f69a01ab11d2ad342f470acf41ca36b3048f6dc854\": rpc error: code = NotFound desc = could not find container \"ca86e1b786959806db89b7f69a01ab11d2ad342f470acf41ca36b3048f6dc854\": container with ID starting with ca86e1b786959806db89b7f69a01ab11d2ad342f470acf41ca36b3048f6dc854 not found: ID does not exist"
Jul 27 19:42:37.855722 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.855699    2562 scope.go:117] "RemoveContainer" containerID="4c2b8b1be5bc4e4a8264fcc71da2570d9e237a2df57428648c4669e2473ae248"
Jul 27 19:42:37.855937 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:42:37.855918    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2b8b1be5bc4e4a8264fcc71da2570d9e237a2df57428648c4669e2473ae248\": container with ID starting with 4c2b8b1be5bc4e4a8264fcc71da2570d9e237a2df57428648c4669e2473ae248 not found: ID does not exist" containerID="4c2b8b1be5bc4e4a8264fcc71da2570d9e237a2df57428648c4669e2473ae248"
Jul 27 19:42:37.856039 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.855939    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2b8b1be5bc4e4a8264fcc71da2570d9e237a2df57428648c4669e2473ae248"} err="failed to get container status \"4c2b8b1be5bc4e4a8264fcc71da2570d9e237a2df57428648c4669e2473ae248\": rpc error: code = NotFound desc = could not find container \"4c2b8b1be5bc4e4a8264fcc71da2570d9e237a2df57428648c4669e2473ae248\": container with ID starting with 4c2b8b1be5bc4e4a8264fcc71da2570d9e237a2df57428648c4669e2473ae248 not found: ID does not exist"
Jul 27 19:42:37.856039 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.855955    2562 scope.go:117] "RemoveContainer" containerID="30c504b419393708502b2e56734d3003f7d0b7b0da40e808030053d25de0678c"
Jul 27 19:42:37.856151 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:42:37.856136    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c504b419393708502b2e56734d3003f7d0b7b0da40e808030053d25de0678c\": container with ID starting with 30c504b419393708502b2e56734d3003f7d0b7b0da40e808030053d25de0678c not found: ID does not exist" containerID="30c504b419393708502b2e56734d3003f7d0b7b0da40e808030053d25de0678c"
Jul 27 19:42:37.856190 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.856155    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c504b419393708502b2e56734d3003f7d0b7b0da40e808030053d25de0678c"} err="failed to get container status \"30c504b419393708502b2e56734d3003f7d0b7b0da40e808030053d25de0678c\": rpc error: code = NotFound desc = could not find container \"30c504b419393708502b2e56734d3003f7d0b7b0da40e808030053d25de0678c\": container with ID starting with 30c504b419393708502b2e56734d3003f7d0b7b0da40e808030053d25de0678c not found: ID does not exist"
Jul 27 19:42:37.856190 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.856167    2562 scope.go:117] "RemoveContainer" containerID="ee24fc90f9cb1b5a434569c3cceb1fe47fe90862a074bcd7b1174e8b4907ea05"
Jul 27 19:42:37.856355 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:42:37.856342    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee24fc90f9cb1b5a434569c3cceb1fe47fe90862a074bcd7b1174e8b4907ea05\": container with ID starting with ee24fc90f9cb1b5a434569c3cceb1fe47fe90862a074bcd7b1174e8b4907ea05 not found: ID does not exist" containerID="ee24fc90f9cb1b5a434569c3cceb1fe47fe90862a074bcd7b1174e8b4907ea05"
Jul 27 19:42:37.856394 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.856357    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee24fc90f9cb1b5a434569c3cceb1fe47fe90862a074bcd7b1174e8b4907ea05"} err="failed to get container status \"ee24fc90f9cb1b5a434569c3cceb1fe47fe90862a074bcd7b1174e8b4907ea05\": rpc error: code = NotFound desc = could not find container \"ee24fc90f9cb1b5a434569c3cceb1fe47fe90862a074bcd7b1174e8b4907ea05\": container with ID starting with ee24fc90f9cb1b5a434569c3cceb1fe47fe90862a074bcd7b1174e8b4907ea05 not found: ID does not exist"
Jul 27 19:42:37.861782 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.861762    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:42:37.862002 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.861991    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="config-reloader"
Jul 27 19:42:37.862037 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862004    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="config-reloader"
Jul 27 19:42:37.862037 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862017    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="kube-rbac-proxy"
Jul 27 19:42:37.862037 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862023    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="kube-rbac-proxy"
Jul 27 19:42:37.862037 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862029    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="thanos-sidecar"
Jul 27 19:42:37.862037 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862034    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="thanos-sidecar"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862040    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="kube-rbac-proxy-thanos"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862045    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="kube-rbac-proxy-thanos"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862052    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="kube-rbac-proxy-web"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862057    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="kube-rbac-proxy-web"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862065    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="init-config-reloader"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862070    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="init-config-reloader"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862076    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be68d266-a08f-475f-ba80-1ec512a2edcd" containerName="console"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862081    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be68d266-a08f-475f-ba80-1ec512a2edcd" containerName="console"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862088    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="prometheus"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862093    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="prometheus"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862133    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="be68d266-a08f-475f-ba80-1ec512a2edcd" containerName="console"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862140    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="config-reloader"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862146    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="prometheus"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862151    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="kube-rbac-proxy-thanos"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862157    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="thanos-sidecar"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862161    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="kube-rbac-proxy-web"
Jul 27 19:42:37.862185 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.862166    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" containerName="kube-rbac-proxy"
Jul 27 19:42:37.867344 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.867328    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.869812 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.869793    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 19:42:37.870377 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.870360    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 19:42:37.870622 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.870607    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cbbwn\""
Jul 27 19:42:37.870735 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.870718    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 19:42:37.870870 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.870827    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 19:42:37.870992 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.870970    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eg6i20k34ans7\""
Jul 27 19:42:37.871094 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.871011    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 19:42:37.871094 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.871040    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 19:42:37.871094 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.871088    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 19:42:37.871325 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.871310    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 19:42:37.871384 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.871343    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 19:42:37.871771 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.871749    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 19:42:37.871862 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.871760    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 19:42:37.871862 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.871794    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 19:42:37.873660 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.873639    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 19:42:37.874115 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.874094    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 19:42:37.877356 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.877333    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 19:42:37.884874 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.884847    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:42:37.920410 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920377    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.920584 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.920584 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920450    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.920584 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.920584 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-web-config\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.920730 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.920730 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920623    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.920730 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920660    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a78bcc2-e19e-45da-946c-5b91c8159602-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.920730 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920676    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-config\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.920730 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920698    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.920730 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.921096 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.921096 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920778    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.921096 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920800    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.921096 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920817    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.921096 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920845    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.921096 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920868    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a78bcc2-e19e-45da-946c-5b91c8159602-config-out\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.921096 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920882    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.921096 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920899    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56hzm\" (UniqueName: \"kubernetes.io/projected/4a78bcc2-e19e-45da-946c-5b91c8159602-kube-api-access-56hzm\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.921096 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.920916    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:37.989894 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:37.989864    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d53ab6e3-7565-40f9-bb2b-6194702d94e3" path="/var/lib/kubelet/pods/d53ab6e3-7565-40f9-bb2b-6194702d94e3/volumes"
Jul 27 19:42:38.022070 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022039    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022070 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022071    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022228 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a78bcc2-e19e-45da-946c-5b91c8159602-config-out\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022228 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022228 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56hzm\" (UniqueName: \"kubernetes.io/projected/4a78bcc2-e19e-45da-946c-5b91c8159602-kube-api-access-56hzm\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022381 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022286    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022381 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022335    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022511 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022387    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022511 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022437    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022511 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022657 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022520    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-web-config\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022657 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022556    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022657 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022657 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022624    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a78bcc2-e19e-45da-946c-5b91c8159602-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022657 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022651    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-config\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022928 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022686    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022928 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022716    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022928 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022757    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022928 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022785    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022928 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022820    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.022928 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.022830    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.024004 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.023948    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.024120 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.024074    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.025161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.024207    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.025161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.024467    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.025161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.025118    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.025353 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.025178    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.026590 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.026546    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.026752 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.026733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.027162 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.027087    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a78bcc2-e19e-45da-946c-5b91c8159602-config-out\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.027533 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.027509    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.027728 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.027701    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-config\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.027854 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.027767    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.028052 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.028030    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.028201 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.028182    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a78bcc2-e19e-45da-946c-5b91c8159602-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.028267 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.028204    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.028348 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.028321    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.028435 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.028364    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a78bcc2-e19e-45da-946c-5b91c8159602-web-config\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.029211 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.029192    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a78bcc2-e19e-45da-946c-5b91c8159602-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.030829 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.030809    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56hzm\" (UniqueName: \"kubernetes.io/projected/4a78bcc2-e19e-45da-946c-5b91c8159602-kube-api-access-56hzm\") pod \"prometheus-k8s-0\" (UID: \"4a78bcc2-e19e-45da-946c-5b91c8159602\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.177141 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.177095    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:42:38.323974 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.323940    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:42:38.327721 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:42:38.327697    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a78bcc2_e19e_45da_946c_5b91c8159602.slice/crio-6b083f4b4b579e314c168e58f52e3955b915ea5f876142b14e78ffb41681291c WatchSource:0}: Error finding container 6b083f4b4b579e314c168e58f52e3955b915ea5f876142b14e78ffb41681291c: Status 404 returned error can't find the container with id 6b083f4b4b579e314c168e58f52e3955b915ea5f876142b14e78ffb41681291c
Jul 27 19:42:38.807574 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.807538    2562 generic.go:356] "Generic (PLEG): container finished" podID="4a78bcc2-e19e-45da-946c-5b91c8159602" containerID="803d1ad3407b76908d5115aa6c7551f0c1871597824cc041402ba5d5b781bc4c" exitCode=0
Jul 27 19:42:38.807755 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.807632    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a78bcc2-e19e-45da-946c-5b91c8159602","Type":"ContainerDied","Data":"803d1ad3407b76908d5115aa6c7551f0c1871597824cc041402ba5d5b781bc4c"}
Jul 27 19:42:38.807755 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:38.807710    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a78bcc2-e19e-45da-946c-5b91c8159602","Type":"ContainerStarted","Data":"6b083f4b4b579e314c168e58f52e3955b915ea5f876142b14e78ffb41681291c"}
Jul 27 19:42:39.815605 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:39.815571    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a78bcc2-e19e-45da-946c-5b91c8159602","Type":"ContainerStarted","Data":"bc3d4961abe72f279357aefca3c1d3489865c9e7802f22fbf601d316e03039ac"}
Jul 27 19:42:39.815956 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:39.815636    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a78bcc2-e19e-45da-946c-5b91c8159602","Type":"ContainerStarted","Data":"72ade9f987bebb52e2f14cf22bab8cfa4b267cec1cc7d5d312fba76bbf45948b"}
Jul 27 19:42:39.815956 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:39.815661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a78bcc2-e19e-45da-946c-5b91c8159602","Type":"ContainerStarted","Data":"d6a16a8b16c3086bc25575a0e79f5722638f0bb4bbaabc49bc6a746be970f208"}
Jul 27 19:42:39.815956 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:39.815684    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a78bcc2-e19e-45da-946c-5b91c8159602","Type":"ContainerStarted","Data":"3a48333734cc4092833472d7da79a9051abf921026c28f3c059b1f740444d44c"}
Jul 27 19:42:39.815956 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:39.815708    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a78bcc2-e19e-45da-946c-5b91c8159602","Type":"ContainerStarted","Data":"8b06e536218e31090e1724574c2838a5104cf39efbd5dd7ee1ce874d922ac114"}
Jul 27 19:42:39.815956 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:39.815732    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a78bcc2-e19e-45da-946c-5b91c8159602","Type":"ContainerStarted","Data":"6b3d19e04c871ae3ae7cb1539814208500f104d0cd8d2eb85843f7bb0e5ce4a0"}
Jul 27 19:42:39.851761 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:39.851708    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.833031918 podStartE2EDuration="2.851692316s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.660398ms" isStatefulPod=false podCreationTimestamp="2026-07-27 19:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:42:39.848745044 +0000 UTC m=+174.400661462" watchObservedRunningTime="2026-07-27 19:42:39.851692316 +0000 UTC m=+174.403608796"
Jul 27 19:42:43.178179 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:42:43.178137    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:43:38.178149 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:43:38.178114    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:43:38.194024 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:43:38.193990    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:43:39.047645 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:43:39.047618    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:44:20.268784 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.268706    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"]
Jul 27 19:44:20.271943 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.271919    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.273954 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.273929    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 19:44:20.274515 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.274470    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 19:44:20.274708 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.274688    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 19:44:20.274821 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.274688    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6clqf\""
Jul 27 19:44:20.276800 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.276784    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 19:44:20.307493 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.307446    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"]
Jul 27 19:44:20.346921 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.346893    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/56f55834-fe4e-403f-a5ad-1d21555238a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.347078 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.346932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56f55834-fe4e-403f-a5ad-1d21555238a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.347078 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.346958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56f55834-fe4e-403f-a5ad-1d21555238a9-tmp\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.347078 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.347059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56f55834-fe4e-403f-a5ad-1d21555238a9-manifests\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.347179 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.347096    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4lmk\" (UniqueName: \"kubernetes.io/projected/56f55834-fe4e-403f-a5ad-1d21555238a9-kube-api-access-r4lmk\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.447424 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.447391    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56f55834-fe4e-403f-a5ad-1d21555238a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.447424 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.447426    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56f55834-fe4e-403f-a5ad-1d21555238a9-tmp\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.447682 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.447497    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56f55834-fe4e-403f-a5ad-1d21555238a9-manifests\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.447682 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.447529    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4lmk\" (UniqueName: \"kubernetes.io/projected/56f55834-fe4e-403f-a5ad-1d21555238a9-kube-api-access-r4lmk\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.447682 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.447555    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/56f55834-fe4e-403f-a5ad-1d21555238a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.447989 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.447966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56f55834-fe4e-403f-a5ad-1d21555238a9-manifests\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.449951 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.449923    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56f55834-fe4e-403f-a5ad-1d21555238a9-tmp\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.450063 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.450026    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/56f55834-fe4e-403f-a5ad-1d21555238a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.450063 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.450038    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56f55834-fe4e-403f-a5ad-1d21555238a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.456534 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.456514    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4lmk\" (UniqueName: \"kubernetes.io/projected/56f55834-fe4e-403f-a5ad-1d21555238a9-kube-api-access-r4lmk\") pod \"opendatahub-operator-controller-manager-7df5588f67-626t9\" (UID: \"56f55834-fe4e-403f-a5ad-1d21555238a9\") " pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.582252 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.582170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:20.729004 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:20.728971    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"]
Jul 27 19:44:20.732180 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:44:20.732153    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f55834_fe4e_403f_a5ad_1d21555238a9.slice/crio-4ca139d89441805eeb09151d135760e2498728d8ac86ae935c10fec5041e1f69 WatchSource:0}: Error finding container 4ca139d89441805eeb09151d135760e2498728d8ac86ae935c10fec5041e1f69: Status 404 returned error can't find the container with id 4ca139d89441805eeb09151d135760e2498728d8ac86ae935c10fec5041e1f69
Jul 27 19:44:21.177391 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:21.177354    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9" event={"ID":"56f55834-fe4e-403f-a5ad-1d21555238a9","Type":"ContainerStarted","Data":"4ca139d89441805eeb09151d135760e2498728d8ac86ae935c10fec5041e1f69"}
Jul 27 19:44:24.192053 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:24.192014    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9" event={"ID":"56f55834-fe4e-403f-a5ad-1d21555238a9","Type":"ContainerStarted","Data":"ff81c89b05001edd7c8078138fc90820ae11c77e24bb30260c45f348d9ee9dbc"}
Jul 27 19:44:25.198842 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:25.198811    2562 generic.go:356] "Generic (PLEG): container finished" podID="56f55834-fe4e-403f-a5ad-1d21555238a9" containerID="ff81c89b05001edd7c8078138fc90820ae11c77e24bb30260c45f348d9ee9dbc" exitCode=0
Jul 27 19:44:25.199176 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:25.198887    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9" event={"ID":"56f55834-fe4e-403f-a5ad-1d21555238a9","Type":"ContainerDied","Data":"ff81c89b05001edd7c8078138fc90820ae11c77e24bb30260c45f348d9ee9dbc"}
Jul 27 19:44:26.204301 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:26.204266    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9" event={"ID":"56f55834-fe4e-403f-a5ad-1d21555238a9","Type":"ContainerStarted","Data":"73458dae3b8e2450b6c8f2e9bb1d6e11fa5d1e96e33fbd05358acac6764e7d91"}
Jul 27 19:44:26.204719 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:26.204338    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:26.236190 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:26.236133    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9" podStartSLOduration=1.910242953 podStartE2EDuration="6.236117773s" totalImagesPullingTime="3.238259138s" totalInitContainerRuntime="1.087615682s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:44:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:44:26.234230912 +0000 UTC m=+280.786147329" watchObservedRunningTime="2026-07-27 19:44:26.236117773 +0000 UTC m=+280.788034249"
Jul 27 19:44:37.210454 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:37.210426    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7df5588f67-626t9"
Jul 27 19:44:39.620037 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.620003    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj"]
Jul 27 19:44:39.625777 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.625761    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj"
Jul 27 19:44:39.633871 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.633849    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 19:44:39.635040 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.635020    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 19:44:39.639598 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.639584    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zq4br\""
Jul 27 19:44:39.639672 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.639591    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 19:44:39.640349 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.640336    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 19:44:39.675893 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.675858    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj"]
Jul 27 19:44:39.678630 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.678609    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8799eeb6-ba19-46de-85e6-007eb62dcf70-tmp\") pod \"kube-auth-proxy-5fd5d97dbc-fllhj\" (UID: \"8799eeb6-ba19-46de-85e6-007eb62dcf70\") " pod="openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj"
Jul 27 19:44:39.678744 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.678648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd2rp\" (UniqueName: \"kubernetes.io/projected/8799eeb6-ba19-46de-85e6-007eb62dcf70-kube-api-access-fd2rp\") pod \"kube-auth-proxy-5fd5d97dbc-fllhj\" (UID: \"8799eeb6-ba19-46de-85e6-007eb62dcf70\") " pod="openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj"
Jul 27 19:44:39.678744 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.678677    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8799eeb6-ba19-46de-85e6-007eb62dcf70-tls-certs\") pod \"kube-auth-proxy-5fd5d97dbc-fllhj\" (UID: \"8799eeb6-ba19-46de-85e6-007eb62dcf70\") " pod="openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj"
Jul 27 19:44:39.779350 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.779314    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8799eeb6-ba19-46de-85e6-007eb62dcf70-tls-certs\") pod \"kube-auth-proxy-5fd5d97dbc-fllhj\" (UID: \"8799eeb6-ba19-46de-85e6-007eb62dcf70\") " pod="openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj"
Jul 27 19:44:39.779537 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.779366    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8799eeb6-ba19-46de-85e6-007eb62dcf70-tmp\") pod \"kube-auth-proxy-5fd5d97dbc-fllhj\" (UID: \"8799eeb6-ba19-46de-85e6-007eb62dcf70\") " pod="openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj"
Jul 27 19:44:39.779537 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.779517    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd2rp\" (UniqueName: \"kubernetes.io/projected/8799eeb6-ba19-46de-85e6-007eb62dcf70-kube-api-access-fd2rp\") pod \"kube-auth-proxy-5fd5d97dbc-fllhj\" (UID: \"8799eeb6-ba19-46de-85e6-007eb62dcf70\") " pod="openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj"
Jul 27 19:44:39.781656 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.781630    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8799eeb6-ba19-46de-85e6-007eb62dcf70-tmp\") pod \"kube-auth-proxy-5fd5d97dbc-fllhj\" (UID: \"8799eeb6-ba19-46de-85e6-007eb62dcf70\") " pod="openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj"
Jul 27 19:44:39.781874 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.781857    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8799eeb6-ba19-46de-85e6-007eb62dcf70-tls-certs\") pod \"kube-auth-proxy-5fd5d97dbc-fllhj\" (UID: \"8799eeb6-ba19-46de-85e6-007eb62dcf70\") " pod="openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj"
Jul 27 19:44:39.795553 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.795532    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-p79zv"]
Jul 27 19:44:39.798385 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.798371    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-p79zv"
Jul 27 19:44:39.801120 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.801102    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-djflq\""
Jul 27 19:44:39.801223 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.801151    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 27 19:44:39.838315 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.838283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd2rp\" (UniqueName: \"kubernetes.io/projected/8799eeb6-ba19-46de-85e6-007eb62dcf70-kube-api-access-fd2rp\") pod \"kube-auth-proxy-5fd5d97dbc-fllhj\" (UID: \"8799eeb6-ba19-46de-85e6-007eb62dcf70\") " pod="openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj"
Jul 27 19:44:39.855365 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.855334    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-p79zv"]
Jul 27 19:44:39.880283 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.880196    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6psv\" (UniqueName: \"kubernetes.io/projected/a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa-kube-api-access-p6psv\") pod \"odh-model-controller-7998c67dc4-p79zv\" (UID: \"a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-p79zv"
Jul 27 19:44:39.880283 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.880239    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa-cert\") pod \"odh-model-controller-7998c67dc4-p79zv\" (UID: \"a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-p79zv"
Jul 27 19:44:39.935079 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.935044    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj"
Jul 27 19:44:39.980872 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.980834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6psv\" (UniqueName: \"kubernetes.io/projected/a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa-kube-api-access-p6psv\") pod \"odh-model-controller-7998c67dc4-p79zv\" (UID: \"a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-p79zv"
Jul 27 19:44:39.981053 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.980892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa-cert\") pod \"odh-model-controller-7998c67dc4-p79zv\" (UID: \"a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-p79zv"
Jul 27 19:44:39.981053 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:44:39.980991    2562 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 27 19:44:39.981162 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:44:39.981062    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa-cert podName:a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa nodeName:}" failed. No retries permitted until 2026-07-27 19:44:40.48104482 +0000 UTC m=+295.032961216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa-cert") pod "odh-model-controller-7998c67dc4-p79zv" (UID: "a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa") : secret "odh-model-controller-webhook-cert" not found
Jul 27 19:44:39.992663 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:39.992638    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6psv\" (UniqueName: \"kubernetes.io/projected/a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa-kube-api-access-p6psv\") pod \"odh-model-controller-7998c67dc4-p79zv\" (UID: \"a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-p79zv"
Jul 27 19:44:40.081009 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.080973    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj"]
Jul 27 19:44:40.084746 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:44:40.084710    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8799eeb6_ba19_46de_85e6_007eb62dcf70.slice/crio-047e28a6a97ab73cd456db0cd3369ac8d290b55bb6468fe3dbc41c131b264024 WatchSource:0}: Error finding container 047e28a6a97ab73cd456db0cd3369ac8d290b55bb6468fe3dbc41c131b264024: Status 404 returned error can't find the container with id 047e28a6a97ab73cd456db0cd3369ac8d290b55bb6468fe3dbc41c131b264024
Jul 27 19:44:40.256265 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.256176    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj" event={"ID":"8799eeb6-ba19-46de-85e6-007eb62dcf70","Type":"ContainerStarted","Data":"047e28a6a97ab73cd456db0cd3369ac8d290b55bb6468fe3dbc41c131b264024"}
Jul 27 19:44:40.485348 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.485306    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa-cert\") pod \"odh-model-controller-7998c67dc4-p79zv\" (UID: \"a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-p79zv"
Jul 27 19:44:40.487770 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.487742    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa-cert\") pod \"odh-model-controller-7998c67dc4-p79zv\" (UID: \"a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-p79zv"
Jul 27 19:44:40.708015 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.707980    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-p79zv"
Jul 27 19:44:40.766303 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.765370    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-shdxq"]
Jul 27 19:44:40.769216 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.769193    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-shdxq"
Jul 27 19:44:40.771618 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.771592    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 27 19:44:40.779861 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.779837    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-shdxq"]
Jul 27 19:44:40.788326 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.788290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ed7694-6b8f-4519-bc40-9ac72f89945a-tls-certs\") pod \"model-serving-api-54d89bd5fc-shdxq\" (UID: \"a7ed7694-6b8f-4519-bc40-9ac72f89945a\") " pod="opendatahub/model-serving-api-54d89bd5fc-shdxq"
Jul 27 19:44:40.788434 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.788379    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2vrj\" (UniqueName: \"kubernetes.io/projected/a7ed7694-6b8f-4519-bc40-9ac72f89945a-kube-api-access-c2vrj\") pod \"model-serving-api-54d89bd5fc-shdxq\" (UID: \"a7ed7694-6b8f-4519-bc40-9ac72f89945a\") " pod="opendatahub/model-serving-api-54d89bd5fc-shdxq"
Jul 27 19:44:40.853218 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.853189    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-p79zv"]
Jul 27 19:44:40.855872 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:44:40.855842    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ee9a3c_80b0_4db1_b61f_f34240e5f2aa.slice/crio-68e1ed1fc74f7ff128479d8623e6bf6f8cb47d916222e316f0928c9a1ce5d54f WatchSource:0}: Error finding container 68e1ed1fc74f7ff128479d8623e6bf6f8cb47d916222e316f0928c9a1ce5d54f: Status 404 returned error can't find the container with id 68e1ed1fc74f7ff128479d8623e6bf6f8cb47d916222e316f0928c9a1ce5d54f
Jul 27 19:44:40.889737 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.889705    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ed7694-6b8f-4519-bc40-9ac72f89945a-tls-certs\") pod \"model-serving-api-54d89bd5fc-shdxq\" (UID: \"a7ed7694-6b8f-4519-bc40-9ac72f89945a\") " pod="opendatahub/model-serving-api-54d89bd5fc-shdxq"
Jul 27 19:44:40.889908 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.889760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2vrj\" (UniqueName: \"kubernetes.io/projected/a7ed7694-6b8f-4519-bc40-9ac72f89945a-kube-api-access-c2vrj\") pod \"model-serving-api-54d89bd5fc-shdxq\" (UID: \"a7ed7694-6b8f-4519-bc40-9ac72f89945a\") " pod="opendatahub/model-serving-api-54d89bd5fc-shdxq"
Jul 27 19:44:40.889908 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:44:40.889871    2562 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 27 19:44:40.890019 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:44:40.889945    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ed7694-6b8f-4519-bc40-9ac72f89945a-tls-certs podName:a7ed7694-6b8f-4519-bc40-9ac72f89945a nodeName:}" failed. No retries permitted until 2026-07-27 19:44:41.389927896 +0000 UTC m=+295.941844293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a7ed7694-6b8f-4519-bc40-9ac72f89945a-tls-certs") pod "model-serving-api-54d89bd5fc-shdxq" (UID: "a7ed7694-6b8f-4519-bc40-9ac72f89945a") : secret "model-serving-api-tls" not found
Jul 27 19:44:40.901553 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.901526    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2vrj\" (UniqueName: \"kubernetes.io/projected/a7ed7694-6b8f-4519-bc40-9ac72f89945a-kube-api-access-c2vrj\") pod \"model-serving-api-54d89bd5fc-shdxq\" (UID: \"a7ed7694-6b8f-4519-bc40-9ac72f89945a\") " pod="opendatahub/model-serving-api-54d89bd5fc-shdxq"
Jul 27 19:44:40.982024 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.981932    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"]
Jul 27 19:44:40.985671 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.985633    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:40.987871 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.987838    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 27 19:44:40.987996 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.987969    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 27 19:44:40.988067 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.987992    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-889c2\""
Jul 27 19:44:40.988117 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.988095    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 19:44:40.988172 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.988095    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 27 19:44:40.996100 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:40.996080    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"]
Jul 27 19:44:41.091491 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.091437    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.091676 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.091500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.091676 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.091547    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.091676 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.091586    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.091676 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.091613    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.091676 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.091633    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.091676 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.091656    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvx25\" (UniqueName: \"kubernetes.io/projected/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-kube-api-access-xvx25\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.192908 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.192866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.192908 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.192912    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.193142 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.192929    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.193142 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.192965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.193142 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.193085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.193142 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.193111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.193142 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.193135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvx25\" (UniqueName: \"kubernetes.io/projected/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-kube-api-access-xvx25\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.193693 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.193645    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.195864 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.195835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.196097 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.196071    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.196182 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.196156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.196245 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.196233    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.217278 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.217251    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.218700 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.218680    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvx25\" (UniqueName: \"kubernetes.io/projected/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-kube-api-access-xvx25\") pod \"istiod-openshift-gateway-7b7995d5cd-55kqc\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.263347 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.263267    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-p79zv" event={"ID":"a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa","Type":"ContainerStarted","Data":"68e1ed1fc74f7ff128479d8623e6bf6f8cb47d916222e316f0928c9a1ce5d54f"}
Jul 27 19:44:41.298598 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.298568    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:41.395015 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.394984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ed7694-6b8f-4519-bc40-9ac72f89945a-tls-certs\") pod \"model-serving-api-54d89bd5fc-shdxq\" (UID: \"a7ed7694-6b8f-4519-bc40-9ac72f89945a\") " pod="opendatahub/model-serving-api-54d89bd5fc-shdxq"
Jul 27 19:44:41.399749 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.399698    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ed7694-6b8f-4519-bc40-9ac72f89945a-tls-certs\") pod \"model-serving-api-54d89bd5fc-shdxq\" (UID: \"a7ed7694-6b8f-4519-bc40-9ac72f89945a\") " pod="opendatahub/model-serving-api-54d89bd5fc-shdxq"
Jul 27 19:44:41.460088 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.459985    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"]
Jul 27 19:44:41.462581 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:44:41.462544    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adb786d_19fb_4205_b3d5_0e8a4efde5b6.slice/crio-2f853850249e638ea4d529ce158ccf2d5c94c7593b30821f43e3d74aee7182b0 WatchSource:0}: Error finding container 2f853850249e638ea4d529ce158ccf2d5c94c7593b30821f43e3d74aee7182b0: Status 404 returned error can't find the container with id 2f853850249e638ea4d529ce158ccf2d5c94c7593b30821f43e3d74aee7182b0
Jul 27 19:44:41.680575 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.680238    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-shdxq"
Jul 27 19:44:41.868500 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:41.868400    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-shdxq"]
Jul 27 19:44:42.268797 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:42.268760    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc" event={"ID":"9adb786d-19fb-4205-b3d5-0e8a4efde5b6","Type":"ContainerStarted","Data":"2f853850249e638ea4d529ce158ccf2d5c94c7593b30821f43e3d74aee7182b0"}
Jul 27 19:44:42.270077 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:42.270035    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-shdxq" event={"ID":"a7ed7694-6b8f-4519-bc40-9ac72f89945a","Type":"ContainerStarted","Data":"0a6f4297f53cb722ff7cbc64d094af8de5228be87466efb0124e8b2a22010d01"}
Jul 27 19:44:44.283596 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:44.283548    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-p79zv" event={"ID":"a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa","Type":"ContainerStarted","Data":"5376084dd234fd3511d4686afd1f60dfdbdc3c9bacb8e1353420438dd55f9b32"}
Jul 27 19:44:44.284440 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:44.284416    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-p79zv"
Jul 27 19:44:44.286332 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:44.286303    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj" event={"ID":"8799eeb6-ba19-46de-85e6-007eb62dcf70","Type":"ContainerStarted","Data":"7b60f9ab7c04e1d100c2eddf17200f8e2480585ff48fefd68b2829a4100bf376"}
Jul 27 19:44:44.323813 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:44.323745    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5fd5d97dbc-fllhj" podStartSLOduration=1.265785416 podStartE2EDuration="5.323723374s" totalImagesPullingTime="4.057937958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:44:44.322582303 +0000 UTC m=+298.874498747" watchObservedRunningTime="2026-07-27 19:44:44.323723374 +0000 UTC m=+298.875639793"
Jul 27 19:44:44.324011 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:44.323865    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-p79zv" podStartSLOduration=2.039191124 podStartE2EDuration="5.323856867s" totalImagesPullingTime="3.284665743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:44:44.303517134 +0000 UTC m=+298.855433553" watchObservedRunningTime="2026-07-27 19:44:44.323856867 +0000 UTC m=+298.875773286"
Jul 27 19:44:45.292789 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:45.292701    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-shdxq" event={"ID":"a7ed7694-6b8f-4519-bc40-9ac72f89945a","Type":"ContainerStarted","Data":"e1a146cd0b57a3a0afd11da53ec3e87ae1e2105b77db66009b1e9d54567f0999"}
Jul 27 19:44:45.311196 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:45.311141    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-shdxq" podStartSLOduration=2.161833065 podStartE2EDuration="5.311123757s" totalImagesPullingTime="3.149290692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:44:45.310365229 +0000 UTC m=+299.862281650" watchObservedRunningTime="2026-07-27 19:44:45.311123757 +0000 UTC m=+299.863040177"
Jul 27 19:44:46.215864 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:46.215840    2562 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 19:44:46.247465 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:46.247431    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 19:44:46.247539 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:46.247515    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 19:44:46.296544 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:46.296520    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-shdxq"
Jul 27 19:44:47.304097 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:47.304052    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc" event={"ID":"9adb786d-19fb-4205-b3d5-0e8a4efde5b6","Type":"ContainerStarted","Data":"84d80de198812e328a480f22e4b81797afd59b17c1b367faec0f8361a2afeb46"}
Jul 27 19:44:47.304589 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:47.304249    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:47.305556 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:47.305525    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-55kqc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 19:44:47.305646 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:47.305584    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc" podUID="9adb786d-19fb-4205-b3d5-0e8a4efde5b6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 19:44:47.330202 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:47.330145    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc" podStartSLOduration=2.54842803 podStartE2EDuration="7.330127022s" totalImagesPullingTime="4.781698992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:44:47.327763007 +0000 UTC m=+301.879679425" watchObservedRunningTime="2026-07-27 19:44:47.330127022 +0000 UTC m=+301.882043432"
Jul 27 19:44:48.308841 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.308795    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:48.333899 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.333855    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"]
Jul 27 19:44:48.337572 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.337552    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.339821 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.339775    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-spv5q\""
Jul 27 19:44:48.357881 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.357811    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"]
Jul 27 19:44:48.466112 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.466080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.466276 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.466116    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.466276 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.466144    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.466276 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.466168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.466276 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.466189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.466276 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.466228    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.466553 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.466313    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngmh9\" (UniqueName: \"kubernetes.io/projected/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-kube-api-access-ngmh9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.466553 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.466361    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.466553 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.466416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.566904 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.566818    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.566904 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.566867    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngmh9\" (UniqueName: \"kubernetes.io/projected/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-kube-api-access-ngmh9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.567126 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.566993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.567126 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.567048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.567126 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.567115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.567280 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.567147    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.567280 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.567177    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.567280 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.567204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.567280 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.567235    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.567280 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.567239    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.567280 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.567269    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.567568 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.567468    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.567635 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.567599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.567836 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.567818    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.569683 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.569659    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.569821 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.569805    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.590415 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.590390    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.590548 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.590530    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngmh9\" (UniqueName: \"kubernetes.io/projected/64c987d6-44b0-4d39-8bb4-a4e5cc37e157-kube-api-access-ngmh9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw\" (UID: \"64c987d6-44b0-4d39-8bb4-a4e5cc37e157\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.648918 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.648886    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:48.828389 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.828285    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"]
Jul 27 19:44:48.833465 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:44:48.833439    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c987d6_44b0_4d39_8bb4_a4e5cc37e157.slice/crio-ebf482d66b05c334ed69b76091402b58ec789d98d789443bcd3bcacfcaf620ce WatchSource:0}: Error finding container ebf482d66b05c334ed69b76091402b58ec789d98d789443bcd3bcacfcaf620ce: Status 404 returned error can't find the container with id ebf482d66b05c334ed69b76091402b58ec789d98d789443bcd3bcacfcaf620ce
Jul 27 19:44:48.835569 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:48.835551    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:44:49.313426 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:49.313391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw" event={"ID":"64c987d6-44b0-4d39-8bb4-a4e5cc37e157","Type":"ContainerStarted","Data":"ebf482d66b05c334ed69b76091402b58ec789d98d789443bcd3bcacfcaf620ce"}
Jul 27 19:44:50.778285 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.778246    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm"]
Jul 27 19:44:50.781735 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.781708    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm"
Jul 27 19:44:50.785320 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.785299    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 27 19:44:50.787868 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.787854    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rvlmd\""
Jul 27 19:44:50.793059 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.793037    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w"]
Jul 27 19:44:50.797888 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.797281    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w"
Jul 27 19:44:50.804434 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.804197    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-zqhtb\""
Jul 27 19:44:50.804434 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.804311    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 27 19:44:50.808558 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.807906    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm"]
Jul 27 19:44:50.812924 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.812898    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w"]
Jul 27 19:44:50.890247 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.890212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21a0d151-3240-46fe-8c19-e64e2d60e684-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bwq5w\" (UID: \"21a0d151-3240-46fe-8c19-e64e2d60e684\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w"
Jul 27 19:44:50.890247 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.890251    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrbtl\" (UniqueName: \"kubernetes.io/projected/21a0d151-3240-46fe-8c19-e64e2d60e684-kube-api-access-qrbtl\") pod \"llmisvc-controller-manager-777c6bd7d5-bwq5w\" (UID: \"21a0d151-3240-46fe-8c19-e64e2d60e684\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w"
Jul 27 19:44:50.890497 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.890423    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbj9r\" (UniqueName: \"kubernetes.io/projected/b4cd6b2f-c1db-4267-a123-ba564753c91d-kube-api-access-fbj9r\") pod \"kserve-controller-manager-d6f4b65b8-pxkfm\" (UID: \"b4cd6b2f-c1db-4267-a123-ba564753c91d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm"
Jul 27 19:44:50.890559 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.890470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4cd6b2f-c1db-4267-a123-ba564753c91d-cert\") pod \"kserve-controller-manager-d6f4b65b8-pxkfm\" (UID: \"b4cd6b2f-c1db-4267-a123-ba564753c91d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm"
Jul 27 19:44:50.991267 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.991234    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbj9r\" (UniqueName: \"kubernetes.io/projected/b4cd6b2f-c1db-4267-a123-ba564753c91d-kube-api-access-fbj9r\") pod \"kserve-controller-manager-d6f4b65b8-pxkfm\" (UID: \"b4cd6b2f-c1db-4267-a123-ba564753c91d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm"
Jul 27 19:44:50.991583 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.991285    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4cd6b2f-c1db-4267-a123-ba564753c91d-cert\") pod \"kserve-controller-manager-d6f4b65b8-pxkfm\" (UID: \"b4cd6b2f-c1db-4267-a123-ba564753c91d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm"
Jul 27 19:44:50.991583 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.991319    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21a0d151-3240-46fe-8c19-e64e2d60e684-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bwq5w\" (UID: \"21a0d151-3240-46fe-8c19-e64e2d60e684\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w"
Jul 27 19:44:50.991583 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.991342    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrbtl\" (UniqueName: \"kubernetes.io/projected/21a0d151-3240-46fe-8c19-e64e2d60e684-kube-api-access-qrbtl\") pod \"llmisvc-controller-manager-777c6bd7d5-bwq5w\" (UID: \"21a0d151-3240-46fe-8c19-e64e2d60e684\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w"
Jul 27 19:44:50.992123 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:44:50.991849    2562 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 27 19:44:50.992123 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:44:50.991919    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a0d151-3240-46fe-8c19-e64e2d60e684-cert podName:21a0d151-3240-46fe-8c19-e64e2d60e684 nodeName:}" failed. No retries permitted until 2026-07-27 19:44:51.491898418 +0000 UTC m=+306.043814813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/21a0d151-3240-46fe-8c19-e64e2d60e684-cert") pod "llmisvc-controller-manager-777c6bd7d5-bwq5w" (UID: "21a0d151-3240-46fe-8c19-e64e2d60e684") : secret "llmisvc-webhook-server-cert" not found
Jul 27 19:44:50.994964 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:50.994937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4cd6b2f-c1db-4267-a123-ba564753c91d-cert\") pod \"kserve-controller-manager-d6f4b65b8-pxkfm\" (UID: \"b4cd6b2f-c1db-4267-a123-ba564753c91d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm"
Jul 27 19:44:51.046097 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:51.046018    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbj9r\" (UniqueName: \"kubernetes.io/projected/b4cd6b2f-c1db-4267-a123-ba564753c91d-kube-api-access-fbj9r\") pod \"kserve-controller-manager-d6f4b65b8-pxkfm\" (UID: \"b4cd6b2f-c1db-4267-a123-ba564753c91d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm"
Jul 27 19:44:51.056285 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:51.056260    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrbtl\" (UniqueName: \"kubernetes.io/projected/21a0d151-3240-46fe-8c19-e64e2d60e684-kube-api-access-qrbtl\") pod \"llmisvc-controller-manager-777c6bd7d5-bwq5w\" (UID: \"21a0d151-3240-46fe-8c19-e64e2d60e684\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w"
Jul 27 19:44:51.097168 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:51.097129    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm"
Jul 27 19:44:51.496800 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:51.496768    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21a0d151-3240-46fe-8c19-e64e2d60e684-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bwq5w\" (UID: \"21a0d151-3240-46fe-8c19-e64e2d60e684\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w"
Jul 27 19:44:51.499616 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:51.499596    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21a0d151-3240-46fe-8c19-e64e2d60e684-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bwq5w\" (UID: \"21a0d151-3240-46fe-8c19-e64e2d60e684\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w"
Jul 27 19:44:51.721719 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:51.721687    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w"
Jul 27 19:44:51.905261 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:51.905212    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 19:44:51.905680 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:51.905316    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 19:44:51.905680 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:51.905367    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 19:44:51.989827 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:44:51.989794    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cd6b2f_c1db_4267_a123_ba564753c91d.slice/crio-1a53662d75fb3656c8fa8fee6fed1092f48786a25f6aa518c39c9bd94d6d5a79 WatchSource:0}: Error finding container 1a53662d75fb3656c8fa8fee6fed1092f48786a25f6aa518c39c9bd94d6d5a79: Status 404 returned error can't find the container with id 1a53662d75fb3656c8fa8fee6fed1092f48786a25f6aa518c39c9bd94d6d5a79
Jul 27 19:44:51.991726 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:51.991600    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm"]
Jul 27 19:44:52.008062 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:52.008040    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w"]
Jul 27 19:44:52.009918 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:44:52.009893    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod21a0d151_3240_46fe_8c19_e64e2d60e684.slice/crio-0020aa2daa41d8da51263a7fe209a747058eae80f3e2fdb05104d0a3b39bb688 WatchSource:0}: Error finding container 0020aa2daa41d8da51263a7fe209a747058eae80f3e2fdb05104d0a3b39bb688: Status 404 returned error can't find the container with id 0020aa2daa41d8da51263a7fe209a747058eae80f3e2fdb05104d0a3b39bb688
Jul 27 19:44:52.328186 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:52.328145    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw" event={"ID":"64c987d6-44b0-4d39-8bb4-a4e5cc37e157","Type":"ContainerStarted","Data":"4454f1572cfe8b690e359e0dca095574ee409e524e19501f3587d7b4fc853186"}
Jul 27 19:44:52.329374 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:52.329353    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm" event={"ID":"b4cd6b2f-c1db-4267-a123-ba564753c91d","Type":"ContainerStarted","Data":"1a53662d75fb3656c8fa8fee6fed1092f48786a25f6aa518c39c9bd94d6d5a79"}
Jul 27 19:44:52.330518 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:52.330497    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w" event={"ID":"21a0d151-3240-46fe-8c19-e64e2d60e684","Type":"ContainerStarted","Data":"0020aa2daa41d8da51263a7fe209a747058eae80f3e2fdb05104d0a3b39bb688"}
Jul 27 19:44:52.350195 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:52.350137    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw" podStartSLOduration=1.280929987 podStartE2EDuration="4.350115196s" totalImagesPullingTime="3.069185209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:44:52.348236067 +0000 UTC m=+306.900152488" watchObservedRunningTime="2026-07-27 19:44:52.350115196 +0000 UTC m=+306.902031618"
Jul 27 19:44:52.509756 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:52.509725    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"]
Jul 27 19:44:52.510004 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:52.509965    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc" podUID="9adb786d-19fb-4205-b3d5-0e8a4efde5b6" containerName="discovery" containerID="cri-o://84d80de198812e328a480f22e4b81797afd59b17c1b367faec0f8361a2afeb46" gracePeriod=30
Jul 27 19:44:52.649207 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:52.649163    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:52.655740 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:52.655713    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:52.889392 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:52.889323    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:53.011418 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.011382    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-cacerts\") pod \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") "
Jul 27 19:44:53.011952 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.011433    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvx25\" (UniqueName: \"kubernetes.io/projected/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-kube-api-access-xvx25\") pod \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") "
Jul 27 19:44:53.011952 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.011469    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-csr-dns-cert\") pod \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") "
Jul 27 19:44:53.011952 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.011534    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-csr-ca-configmap\") pod \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") "
Jul 27 19:44:53.011952 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.011567    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-token\") pod \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") "
Jul 27 19:44:53.011952 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.011589    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-local-certs\") pod \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") "
Jul 27 19:44:53.011952 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.011640    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-kubeconfig\") pod \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\" (UID: \"9adb786d-19fb-4205-b3d5-0e8a4efde5b6\") "
Jul 27 19:44:53.012277 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.012130    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9adb786d-19fb-4205-b3d5-0e8a4efde5b6" (UID: "9adb786d-19fb-4205-b3d5-0e8a4efde5b6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:44:53.015256 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.015215    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9adb786d-19fb-4205-b3d5-0e8a4efde5b6" (UID: "9adb786d-19fb-4205-b3d5-0e8a4efde5b6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:44:53.015797 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.015728    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "9adb786d-19fb-4205-b3d5-0e8a4efde5b6" (UID: "9adb786d-19fb-4205-b3d5-0e8a4efde5b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:44:53.017318 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.017244    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-cacerts" (OuterVolumeSpecName: "cacerts") pod "9adb786d-19fb-4205-b3d5-0e8a4efde5b6" (UID: "9adb786d-19fb-4205-b3d5-0e8a4efde5b6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:44:53.017659 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.017632    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-kube-api-access-xvx25" (OuterVolumeSpecName: "kube-api-access-xvx25") pod "9adb786d-19fb-4205-b3d5-0e8a4efde5b6" (UID: "9adb786d-19fb-4205-b3d5-0e8a4efde5b6"). InnerVolumeSpecName "kube-api-access-xvx25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:44:53.017659 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.017645    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9adb786d-19fb-4205-b3d5-0e8a4efde5b6" (UID: "9adb786d-19fb-4205-b3d5-0e8a4efde5b6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:44:53.018951 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.018919    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-local-certs" (OuterVolumeSpecName: "local-certs") pod "9adb786d-19fb-4205-b3d5-0e8a4efde5b6" (UID: "9adb786d-19fb-4205-b3d5-0e8a4efde5b6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:44:53.112910 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.112872    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvx25\" (UniqueName: \"kubernetes.io/projected/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-kube-api-access-xvx25\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:44:53.112910 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.112911    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-csr-dns-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:44:53.113105 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.112925    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-csr-ca-configmap\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:44:53.113105 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.112941    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:44:53.113105 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.112953    2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-local-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:44:53.113105 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.112966    2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-istio-kubeconfig\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:44:53.113105 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.112980    2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9adb786d-19fb-4205-b3d5-0e8a4efde5b6-cacerts\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:44:53.337412 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.337323    2562 generic.go:356] "Generic (PLEG): container finished" podID="9adb786d-19fb-4205-b3d5-0e8a4efde5b6" containerID="84d80de198812e328a480f22e4b81797afd59b17c1b367faec0f8361a2afeb46" exitCode=0
Jul 27 19:44:53.337412 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.337384    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"
Jul 27 19:44:53.337672 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.337416    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc" event={"ID":"9adb786d-19fb-4205-b3d5-0e8a4efde5b6","Type":"ContainerDied","Data":"84d80de198812e328a480f22e4b81797afd59b17c1b367faec0f8361a2afeb46"}
Jul 27 19:44:53.337672 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.337463    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc" event={"ID":"9adb786d-19fb-4205-b3d5-0e8a4efde5b6","Type":"ContainerDied","Data":"2f853850249e638ea4d529ce158ccf2d5c94c7593b30821f43e3d74aee7182b0"}
Jul 27 19:44:53.337672 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.337506    2562 scope.go:117] "RemoveContainer" containerID="84d80de198812e328a480f22e4b81797afd59b17c1b367faec0f8361a2afeb46"
Jul 27 19:44:53.338027 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.338006    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:53.339037 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.339020    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw"
Jul 27 19:44:53.414347 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.414304    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"]
Jul 27 19:44:53.425682 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.425650    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-55kqc"]
Jul 27 19:44:53.447883 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.447861    2562 scope.go:117] "RemoveContainer" containerID="84d80de198812e328a480f22e4b81797afd59b17c1b367faec0f8361a2afeb46"
Jul 27 19:44:53.448279 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:44:53.448249    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d80de198812e328a480f22e4b81797afd59b17c1b367faec0f8361a2afeb46\": container with ID starting with 84d80de198812e328a480f22e4b81797afd59b17c1b367faec0f8361a2afeb46 not found: ID does not exist" containerID="84d80de198812e328a480f22e4b81797afd59b17c1b367faec0f8361a2afeb46"
Jul 27 19:44:53.448398 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.448289    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d80de198812e328a480f22e4b81797afd59b17c1b367faec0f8361a2afeb46"} err="failed to get container status \"84d80de198812e328a480f22e4b81797afd59b17c1b367faec0f8361a2afeb46\": rpc error: code = NotFound desc = could not find container \"84d80de198812e328a480f22e4b81797afd59b17c1b367faec0f8361a2afeb46\": container with ID starting with 84d80de198812e328a480f22e4b81797afd59b17c1b367faec0f8361a2afeb46 not found: ID does not exist"
Jul 27 19:44:53.994444 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:53.994407    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9adb786d-19fb-4205-b3d5-0e8a4efde5b6" path="/var/lib/kubelet/pods/9adb786d-19fb-4205-b3d5-0e8a4efde5b6/volumes"
Jul 27 19:44:55.350210 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:55.350117    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm" event={"ID":"b4cd6b2f-c1db-4267-a123-ba564753c91d","Type":"ContainerStarted","Data":"8a184f307c71ce408ebc3f5988f52456d28afdffc3ffd0f8c6a89d5bf6de5d27"}
Jul 27 19:44:55.350669 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:55.350249    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm"
Jul 27 19:44:55.351782 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:55.351758    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w" event={"ID":"21a0d151-3240-46fe-8c19-e64e2d60e684","Type":"ContainerStarted","Data":"28056f9e1dbda4ebce89a64394b3c063d40ebfca0ed9b470d1395aea34773a0d"}
Jul 27 19:44:55.378365 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:55.378313    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm" podStartSLOduration=2.299206791 podStartE2EDuration="5.378298884s" totalImagesPullingTime="3.079092093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:44:55.375921778 +0000 UTC m=+309.927838196" watchObservedRunningTime="2026-07-27 19:44:55.378298884 +0000 UTC m=+309.930215302"
Jul 27 19:44:55.399298 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:55.399238    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w" podStartSLOduration=2.333961118 podStartE2EDuration="5.399217819s" totalImagesPullingTime="3.065256701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:44:55.397260636 +0000 UTC m=+309.949177057" watchObservedRunningTime="2026-07-27 19:44:55.399217819 +0000 UTC m=+309.951134240"
Jul 27 19:44:56.298510 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:56.298459    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-p79zv"
Jul 27 19:44:56.356814 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:56.356780    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w"
Jul 27 19:44:57.308110 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:44:57.308080    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-shdxq"
Jul 27 19:45:26.362012 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:26.361982    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxkfm"
Jul 27 19:45:27.365056 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:27.365027    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bwq5w"
Jul 27 19:45:40.422597 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.422514    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-stn49"]
Jul 27 19:45:40.422997 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.422980    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adb786d-19fb-4205-b3d5-0e8a4efde5b6" containerName="discovery"
Jul 27 19:45:40.423039 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.423000    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adb786d-19fb-4205-b3d5-0e8a4efde5b6" containerName="discovery"
Jul 27 19:45:40.423081 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.423061    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adb786d-19fb-4205-b3d5-0e8a4efde5b6" containerName="discovery"
Jul 27 19:45:40.426091 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.426068    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-stn49"
Jul 27 19:45:40.428139 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.428117    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-bm96v\""
Jul 27 19:45:40.428313 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.428294    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 19:45:40.428313 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.428292    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 19:45:40.435609 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.435585    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-stn49"]
Jul 27 19:45:40.517960 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.517924    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jbsj\" (UniqueName: \"kubernetes.io/projected/e35f96be-c6c9-4c86-a899-234573e74e30-kube-api-access-2jbsj\") pod \"kuadrant-operator-catalog-stn49\" (UID: \"e35f96be-c6c9-4c86-a899-234573e74e30\") " pod="kuadrant-system/kuadrant-operator-catalog-stn49"
Jul 27 19:45:40.618538 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.618501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jbsj\" (UniqueName: \"kubernetes.io/projected/e35f96be-c6c9-4c86-a899-234573e74e30-kube-api-access-2jbsj\") pod \"kuadrant-operator-catalog-stn49\" (UID: \"e35f96be-c6c9-4c86-a899-234573e74e30\") " pod="kuadrant-system/kuadrant-operator-catalog-stn49"
Jul 27 19:45:40.626710 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.626680    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jbsj\" (UniqueName: \"kubernetes.io/projected/e35f96be-c6c9-4c86-a899-234573e74e30-kube-api-access-2jbsj\") pod \"kuadrant-operator-catalog-stn49\" (UID: \"e35f96be-c6c9-4c86-a899-234573e74e30\") " pod="kuadrant-system/kuadrant-operator-catalog-stn49"
Jul 27 19:45:40.737788 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.737690    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-stn49"
Jul 27 19:45:40.782561 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.782528    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-stn49"]
Jul 27 19:45:40.865411 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.865381    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-stn49"]
Jul 27 19:45:40.868840 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:45:40.868809    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35f96be_c6c9_4c86_a899_234573e74e30.slice/crio-718e65963add0ba15ef73fff962bc2cca06d4d73275ddca101710c1c434dcdeb WatchSource:0}: Error finding container 718e65963add0ba15ef73fff962bc2cca06d4d73275ddca101710c1c434dcdeb: Status 404 returned error can't find the container with id 718e65963add0ba15ef73fff962bc2cca06d4d73275ddca101710c1c434dcdeb
Jul 27 19:45:40.993820 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.993750    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pvj4x"]
Jul 27 19:45:40.998164 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:40.998149    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pvj4x"
Jul 27 19:45:41.005012 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:41.004980    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pvj4x"]
Jul 27 19:45:41.022960 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:41.022937    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qs2h\" (UniqueName: \"kubernetes.io/projected/8abd6d76-25d6-4667-a7d2-051a753e8972-kube-api-access-8qs2h\") pod \"kuadrant-operator-catalog-pvj4x\" (UID: \"8abd6d76-25d6-4667-a7d2-051a753e8972\") " pod="kuadrant-system/kuadrant-operator-catalog-pvj4x"
Jul 27 19:45:41.124147 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:41.124113    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qs2h\" (UniqueName: \"kubernetes.io/projected/8abd6d76-25d6-4667-a7d2-051a753e8972-kube-api-access-8qs2h\") pod \"kuadrant-operator-catalog-pvj4x\" (UID: \"8abd6d76-25d6-4667-a7d2-051a753e8972\") " pod="kuadrant-system/kuadrant-operator-catalog-pvj4x"
Jul 27 19:45:41.132113 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:41.132090    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qs2h\" (UniqueName: \"kubernetes.io/projected/8abd6d76-25d6-4667-a7d2-051a753e8972-kube-api-access-8qs2h\") pod \"kuadrant-operator-catalog-pvj4x\" (UID: \"8abd6d76-25d6-4667-a7d2-051a753e8972\") " pod="kuadrant-system/kuadrant-operator-catalog-pvj4x"
Jul 27 19:45:41.309093 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:41.309002    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pvj4x"
Jul 27 19:45:41.446526 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:41.446463    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pvj4x"]
Jul 27 19:45:41.448904 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:45:41.448878    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abd6d76_25d6_4667_a7d2_051a753e8972.slice/crio-d351d779e5bca2a7a85be9e327ac9666f652fefaa2a0a26dba914aa95f1b6646 WatchSource:0}: Error finding container d351d779e5bca2a7a85be9e327ac9666f652fefaa2a0a26dba914aa95f1b6646: Status 404 returned error can't find the container with id d351d779e5bca2a7a85be9e327ac9666f652fefaa2a0a26dba914aa95f1b6646
Jul 27 19:45:41.548379 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:41.548349    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-stn49" event={"ID":"e35f96be-c6c9-4c86-a899-234573e74e30","Type":"ContainerStarted","Data":"718e65963add0ba15ef73fff962bc2cca06d4d73275ddca101710c1c434dcdeb"}
Jul 27 19:45:41.549570 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:41.549542    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pvj4x" event={"ID":"8abd6d76-25d6-4667-a7d2-051a753e8972","Type":"ContainerStarted","Data":"d351d779e5bca2a7a85be9e327ac9666f652fefaa2a0a26dba914aa95f1b6646"}
Jul 27 19:45:43.560383 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:43.560254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-stn49" event={"ID":"e35f96be-c6c9-4c86-a899-234573e74e30","Type":"ContainerStarted","Data":"8796a52f8ed92d67cc504a3a47679a679191ebd3ba7efe0c5ae831ac787ecc82"}
Jul 27 19:45:43.560383 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:43.560295    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-stn49" podUID="e35f96be-c6c9-4c86-a899-234573e74e30" containerName="registry-server" containerID="cri-o://8796a52f8ed92d67cc504a3a47679a679191ebd3ba7efe0c5ae831ac787ecc82" gracePeriod=2
Jul 27 19:45:43.561950 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:43.561918    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pvj4x" event={"ID":"8abd6d76-25d6-4667-a7d2-051a753e8972","Type":"ContainerStarted","Data":"6e0d236c52c7798856b3960023b5763dd399805b7df12bf8e94cf3761acb95c0"}
Jul 27 19:45:43.577055 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:43.577013    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-stn49" podStartSLOduration=1.209633436 podStartE2EDuration="3.576994229s" totalImagesPullingTime="2.367360793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:45:43.576272256 +0000 UTC m=+358.128188674" watchObservedRunningTime="2026-07-27 19:45:43.576994229 +0000 UTC m=+358.128910648"
Jul 27 19:45:43.590976 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:43.590932    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-pvj4x" podStartSLOduration=1.800750656 podStartE2EDuration="3.590918276s" totalImagesPullingTime="1.79016762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:45:43.590398971 +0000 UTC m=+358.142315389" watchObservedRunningTime="2026-07-27 19:45:43.590918276 +0000 UTC m=+358.142834719"
Jul 27 19:45:43.807594 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:43.807570    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-stn49"
Jul 27 19:45:43.849503 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:43.849402    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jbsj\" (UniqueName: \"kubernetes.io/projected/e35f96be-c6c9-4c86-a899-234573e74e30-kube-api-access-2jbsj\") pod \"e35f96be-c6c9-4c86-a899-234573e74e30\" (UID: \"e35f96be-c6c9-4c86-a899-234573e74e30\") "
Jul 27 19:45:43.851664 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:43.851639    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35f96be-c6c9-4c86-a899-234573e74e30-kube-api-access-2jbsj" (OuterVolumeSpecName: "kube-api-access-2jbsj") pod "e35f96be-c6c9-4c86-a899-234573e74e30" (UID: "e35f96be-c6c9-4c86-a899-234573e74e30"). InnerVolumeSpecName "kube-api-access-2jbsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:45:43.950843 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:43.950807    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jbsj\" (UniqueName: \"kubernetes.io/projected/e35f96be-c6c9-4c86-a899-234573e74e30-kube-api-access-2jbsj\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:45:44.567636 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:44.567548    2562 generic.go:356] "Generic (PLEG): container finished" podID="e35f96be-c6c9-4c86-a899-234573e74e30" containerID="8796a52f8ed92d67cc504a3a47679a679191ebd3ba7efe0c5ae831ac787ecc82" exitCode=0
Jul 27 19:45:44.567636 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:44.567612    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-stn49"
Jul 27 19:45:44.568101 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:44.567643    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-stn49" event={"ID":"e35f96be-c6c9-4c86-a899-234573e74e30","Type":"ContainerDied","Data":"8796a52f8ed92d67cc504a3a47679a679191ebd3ba7efe0c5ae831ac787ecc82"}
Jul 27 19:45:44.568101 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:44.567697    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-stn49" event={"ID":"e35f96be-c6c9-4c86-a899-234573e74e30","Type":"ContainerDied","Data":"718e65963add0ba15ef73fff962bc2cca06d4d73275ddca101710c1c434dcdeb"}
Jul 27 19:45:44.568101 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:44.567723    2562 scope.go:117] "RemoveContainer" containerID="8796a52f8ed92d67cc504a3a47679a679191ebd3ba7efe0c5ae831ac787ecc82"
Jul 27 19:45:44.576645 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:44.576629    2562 scope.go:117] "RemoveContainer" containerID="8796a52f8ed92d67cc504a3a47679a679191ebd3ba7efe0c5ae831ac787ecc82"
Jul 27 19:45:44.576913 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:45:44.576892    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8796a52f8ed92d67cc504a3a47679a679191ebd3ba7efe0c5ae831ac787ecc82\": container with ID starting with 8796a52f8ed92d67cc504a3a47679a679191ebd3ba7efe0c5ae831ac787ecc82 not found: ID does not exist" containerID="8796a52f8ed92d67cc504a3a47679a679191ebd3ba7efe0c5ae831ac787ecc82"
Jul 27 19:45:44.576984 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:44.576923    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8796a52f8ed92d67cc504a3a47679a679191ebd3ba7efe0c5ae831ac787ecc82"} err="failed to get container status \"8796a52f8ed92d67cc504a3a47679a679191ebd3ba7efe0c5ae831ac787ecc82\": rpc error: code = NotFound desc = could not find container \"8796a52f8ed92d67cc504a3a47679a679191ebd3ba7efe0c5ae831ac787ecc82\": container with ID starting with 8796a52f8ed92d67cc504a3a47679a679191ebd3ba7efe0c5ae831ac787ecc82 not found: ID does not exist"
Jul 27 19:45:44.582260 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:44.582239    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-stn49"]
Jul 27 19:45:44.585812 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:44.585793    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-stn49"]
Jul 27 19:45:45.992059 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:45.992030    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e35f96be-c6c9-4c86-a899-234573e74e30" path="/var/lib/kubelet/pods/e35f96be-c6c9-4c86-a899-234573e74e30/volumes"
Jul 27 19:45:51.309437 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:51.309398    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-pvj4x"
Jul 27 19:45:51.309955 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:51.309460    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-pvj4x"
Jul 27 19:45:51.331053 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:51.331025    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-pvj4x"
Jul 27 19:45:51.620299 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:45:51.620274    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-pvj4x"
Jul 27 19:46:13.788500 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:13.788446    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"]
Jul 27 19:46:13.788973 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:13.788787    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e35f96be-c6c9-4c86-a899-234573e74e30" containerName="registry-server"
Jul 27 19:46:13.788973 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:13.788798    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e35f96be-c6c9-4c86-a899-234573e74e30" containerName="registry-server"
Jul 27 19:46:13.788973 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:13.788873    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e35f96be-c6c9-4c86-a899-234573e74e30" containerName="registry-server"
Jul 27 19:46:13.793857 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:13.793837    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:13.795709 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:13.795689    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-c5b2b\""
Jul 27 19:46:13.806272 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:13.806251    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"]
Jul 27 19:46:13.908619 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:13.908574    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzmvj\" (UniqueName: \"kubernetes.io/projected/ad24108b-7132-4567-8410-c0dc7815ccb9-kube-api-access-lzmvj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\" (UID: \"ad24108b-7132-4567-8410-c0dc7815ccb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:13.908796 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:13.908650    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad24108b-7132-4567-8410-c0dc7815ccb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\" (UID: \"ad24108b-7132-4567-8410-c0dc7815ccb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:14.009964 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:14.009919    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad24108b-7132-4567-8410-c0dc7815ccb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\" (UID: \"ad24108b-7132-4567-8410-c0dc7815ccb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:14.010142 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:14.010016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzmvj\" (UniqueName: \"kubernetes.io/projected/ad24108b-7132-4567-8410-c0dc7815ccb9-kube-api-access-lzmvj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\" (UID: \"ad24108b-7132-4567-8410-c0dc7815ccb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:14.010296 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:14.010276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad24108b-7132-4567-8410-c0dc7815ccb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\" (UID: \"ad24108b-7132-4567-8410-c0dc7815ccb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:14.022740 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:14.022714    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzmvj\" (UniqueName: \"kubernetes.io/projected/ad24108b-7132-4567-8410-c0dc7815ccb9-kube-api-access-lzmvj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\" (UID: \"ad24108b-7132-4567-8410-c0dc7815ccb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:14.103331 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:14.103253    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:14.241669 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:14.241645    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"]
Jul 27 19:46:14.243882 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:46:14.243846    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice/crio-f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67 WatchSource:0}: Error finding container f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67: Status 404 returned error can't find the container with id f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67
Jul 27 19:46:14.698509 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:14.698462    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd" event={"ID":"ad24108b-7132-4567-8410-c0dc7815ccb9","Type":"ContainerStarted","Data":"f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67"}
Jul 27 19:46:15.788323 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:15.785832    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"]
Jul 27 19:46:15.790678 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:15.790649    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:15.793014 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:15.792993    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qgvfw\""
Jul 27 19:46:15.797390 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:15.797355    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"]
Jul 27 19:46:15.926519 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:15.926453    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5zjv\" (UniqueName: \"kubernetes.io/projected/207e141f-84b7-4144-87b6-745374014cc8-kube-api-access-c5zjv\") pod \"limitador-operator-controller-manager-859d7b78bd-nz2g5\" (UID: \"207e141f-84b7-4144-87b6-745374014cc8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:16.027047 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:16.026999    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zjv\" (UniqueName: \"kubernetes.io/projected/207e141f-84b7-4144-87b6-745374014cc8-kube-api-access-c5zjv\") pod \"limitador-operator-controller-manager-859d7b78bd-nz2g5\" (UID: \"207e141f-84b7-4144-87b6-745374014cc8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:16.037179 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:16.037130    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zjv\" (UniqueName: \"kubernetes.io/projected/207e141f-84b7-4144-87b6-745374014cc8-kube-api-access-c5zjv\") pod \"limitador-operator-controller-manager-859d7b78bd-nz2g5\" (UID: \"207e141f-84b7-4144-87b6-745374014cc8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:16.102359 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:16.102240    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:16.256895 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:16.256858    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"]
Jul 27 19:46:16.261527 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:46:16.261467    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice/crio-13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8 WatchSource:0}: Error finding container 13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8: Status 404 returned error can't find the container with id 13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8
Jul 27 19:46:16.710245 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:16.710203    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5" event={"ID":"207e141f-84b7-4144-87b6-745374014cc8","Type":"ContainerStarted","Data":"13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8"}
Jul 27 19:46:18.284030 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:18.283997    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-8hvxh"]
Jul 27 19:46:18.287457 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:18.287440    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8hvxh"
Jul 27 19:46:18.289562 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:18.289540    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-whgw9\""
Jul 27 19:46:18.297091 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:18.297070    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-8hvxh"]
Jul 27 19:46:18.450318 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:18.450283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mwvj\" (UniqueName: \"kubernetes.io/projected/6c006c97-5e13-4121-9ae8-433fdbae3781-kube-api-access-2mwvj\") pod \"authorino-operator-75cc8d5cb7-8hvxh\" (UID: \"6c006c97-5e13-4121-9ae8-433fdbae3781\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-8hvxh"
Jul 27 19:46:18.551228 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:18.551150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mwvj\" (UniqueName: \"kubernetes.io/projected/6c006c97-5e13-4121-9ae8-433fdbae3781-kube-api-access-2mwvj\") pod \"authorino-operator-75cc8d5cb7-8hvxh\" (UID: \"6c006c97-5e13-4121-9ae8-433fdbae3781\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-8hvxh"
Jul 27 19:46:18.563968 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:18.563937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mwvj\" (UniqueName: \"kubernetes.io/projected/6c006c97-5e13-4121-9ae8-433fdbae3781-kube-api-access-2mwvj\") pod \"authorino-operator-75cc8d5cb7-8hvxh\" (UID: \"6c006c97-5e13-4121-9ae8-433fdbae3781\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-8hvxh"
Jul 27 19:46:18.598587 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:18.598550    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8hvxh"
Jul 27 19:46:18.721353 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:18.721318    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5" event={"ID":"207e141f-84b7-4144-87b6-745374014cc8","Type":"ContainerStarted","Data":"f58bc781bdd815c4092c251785dad0eab1956866ee21055104ffc1c9c5f9ac48"}
Jul 27 19:46:18.721565 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:18.721444    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:18.739073 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:18.739014    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5" podStartSLOduration=2.209116809 podStartE2EDuration="3.73899375s" totalImagesPullingTime="1.529876941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:46:18.73858387 +0000 UTC m=+393.290500287" watchObservedRunningTime="2026-07-27 19:46:18.73899375 +0000 UTC m=+393.290910169"
Jul 27 19:46:19.806174 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:19.806151    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-8hvxh"]
Jul 27 19:46:19.808801 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:46:19.808779    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c006c97_5e13_4121_9ae8_433fdbae3781.slice/crio-462104fd521a52bfb00d0eb6a53078962a62aa61fad369dc1a40abb84d4896e3 WatchSource:0}: Error finding container 462104fd521a52bfb00d0eb6a53078962a62aa61fad369dc1a40abb84d4896e3: Status 404 returned error can't find the container with id 462104fd521a52bfb00d0eb6a53078962a62aa61fad369dc1a40abb84d4896e3
Jul 27 19:46:20.735743 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:20.735700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd" event={"ID":"ad24108b-7132-4567-8410-c0dc7815ccb9","Type":"ContainerStarted","Data":"c6ea806639af4bc4541ea2bcce49d8345d1f63ec75665295d1badeaaeb4a7edb"}
Jul 27 19:46:20.735956 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:20.735879    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:20.737289 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:20.737255    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8hvxh" event={"ID":"6c006c97-5e13-4121-9ae8-433fdbae3781","Type":"ContainerStarted","Data":"462104fd521a52bfb00d0eb6a53078962a62aa61fad369dc1a40abb84d4896e3"}
Jul 27 19:46:21.653958 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:21.653895    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd" podStartSLOduration=3.165013678 podStartE2EDuration="8.653870576s" totalImagesPullingTime="5.488856898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:46:20.761578462 +0000 UTC m=+395.313494885" watchObservedRunningTime="2026-07-27 19:46:21.653870576 +0000 UTC m=+396.205786995"
Jul 27 19:46:21.655974 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:21.655946    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dgvzl"]
Jul 27 19:46:21.661061 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:21.661040    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dgvzl"
Jul 27 19:46:21.663275 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:21.663245    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 27 19:46:21.663395 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:21.663326    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-r4x5g\""
Jul 27 19:46:21.674670 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:21.674642    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dgvzl"]
Jul 27 19:46:21.675105 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:21.675082    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqvnc\" (UniqueName: \"kubernetes.io/projected/f9938b9c-d76a-4c14-b8e1-95dad0295fea-kube-api-access-rqvnc\") pod \"dns-operator-controller-manager-864dd78c4c-dgvzl\" (UID: \"f9938b9c-d76a-4c14-b8e1-95dad0295fea\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dgvzl"
Jul 27 19:46:21.776062 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:21.776029    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqvnc\" (UniqueName: \"kubernetes.io/projected/f9938b9c-d76a-4c14-b8e1-95dad0295fea-kube-api-access-rqvnc\") pod \"dns-operator-controller-manager-864dd78c4c-dgvzl\" (UID: \"f9938b9c-d76a-4c14-b8e1-95dad0295fea\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dgvzl"
Jul 27 19:46:21.790393 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:21.790356    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqvnc\" (UniqueName: \"kubernetes.io/projected/f9938b9c-d76a-4c14-b8e1-95dad0295fea-kube-api-access-rqvnc\") pod \"dns-operator-controller-manager-864dd78c4c-dgvzl\" (UID: \"f9938b9c-d76a-4c14-b8e1-95dad0295fea\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dgvzl"
Jul 27 19:46:21.975553 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:21.975462    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dgvzl"
Jul 27 19:46:22.205673 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:22.205648    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dgvzl"]
Jul 27 19:46:22.207891 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:46:22.207863    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9938b9c_d76a_4c14_b8e1_95dad0295fea.slice/crio-b951ffd1f56ce62d26a94c040f45763b4457e907f450717ec7e494fe9ab4605e WatchSource:0}: Error finding container b951ffd1f56ce62d26a94c040f45763b4457e907f450717ec7e494fe9ab4605e: Status 404 returned error can't find the container with id b951ffd1f56ce62d26a94c040f45763b4457e907f450717ec7e494fe9ab4605e
Jul 27 19:46:22.750548 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:22.750511    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8hvxh" event={"ID":"6c006c97-5e13-4121-9ae8-433fdbae3781","Type":"ContainerStarted","Data":"b0b14dd60054ddd9688350ea39173618020868adfad5141c2732583d479bacc5"}
Jul 27 19:46:22.751039 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:22.750611    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8hvxh"
Jul 27 19:46:22.751805 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:22.751785    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dgvzl" event={"ID":"f9938b9c-d76a-4c14-b8e1-95dad0295fea","Type":"ContainerStarted","Data":"b951ffd1f56ce62d26a94c040f45763b4457e907f450717ec7e494fe9ab4605e"}
Jul 27 19:46:22.769379 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:22.769326    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8hvxh" podStartSLOduration=2.461258966 podStartE2EDuration="4.769307391s" totalImagesPullingTime="2.308048425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:46:22.768868111 +0000 UTC m=+397.320784554" watchObservedRunningTime="2026-07-27 19:46:22.769307391 +0000 UTC m=+397.321223809"
Jul 27 19:46:23.010374 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.010299    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"]
Jul 27 19:46:23.014513 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.014471    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"
Jul 27 19:46:23.016832 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.016810    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 27 19:46:23.016965 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.016898    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9przh\""
Jul 27 19:46:23.016965 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.016821    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 27 19:46:23.022758 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.022722    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"]
Jul 27 19:46:23.090082 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.090043    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12c1ad92-cf64-47f8-aff2-3f1a9aebab7d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-f6kfh\" (UID: \"12c1ad92-cf64-47f8-aff2-3f1a9aebab7d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"
Jul 27 19:46:23.090971 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.090291    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pktv\" (UniqueName: \"kubernetes.io/projected/12c1ad92-cf64-47f8-aff2-3f1a9aebab7d-kube-api-access-5pktv\") pod \"kuadrant-console-plugin-66b5774985-f6kfh\" (UID: \"12c1ad92-cf64-47f8-aff2-3f1a9aebab7d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"
Jul 27 19:46:23.090971 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.090355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/12c1ad92-cf64-47f8-aff2-3f1a9aebab7d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-f6kfh\" (UID: \"12c1ad92-cf64-47f8-aff2-3f1a9aebab7d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"
Jul 27 19:46:23.191987 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.191948    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pktv\" (UniqueName: \"kubernetes.io/projected/12c1ad92-cf64-47f8-aff2-3f1a9aebab7d-kube-api-access-5pktv\") pod \"kuadrant-console-plugin-66b5774985-f6kfh\" (UID: \"12c1ad92-cf64-47f8-aff2-3f1a9aebab7d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"
Jul 27 19:46:23.192163 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.191999    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/12c1ad92-cf64-47f8-aff2-3f1a9aebab7d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-f6kfh\" (UID: \"12c1ad92-cf64-47f8-aff2-3f1a9aebab7d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"
Jul 27 19:46:23.192163 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:23.192141    2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 27 19:46:23.192275 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:23.192206    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12c1ad92-cf64-47f8-aff2-3f1a9aebab7d-plugin-serving-cert podName:12c1ad92-cf64-47f8-aff2-3f1a9aebab7d nodeName:}" failed. No retries permitted until 2026-07-27 19:46:23.692185015 +0000 UTC m=+398.244101431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/12c1ad92-cf64-47f8-aff2-3f1a9aebab7d-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-f6kfh" (UID: "12c1ad92-cf64-47f8-aff2-3f1a9aebab7d") : secret "plugin-serving-cert" not found
Jul 27 19:46:23.192349 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.192313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12c1ad92-cf64-47f8-aff2-3f1a9aebab7d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-f6kfh\" (UID: \"12c1ad92-cf64-47f8-aff2-3f1a9aebab7d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"
Jul 27 19:46:23.193100 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.193072    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12c1ad92-cf64-47f8-aff2-3f1a9aebab7d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-f6kfh\" (UID: \"12c1ad92-cf64-47f8-aff2-3f1a9aebab7d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"
Jul 27 19:46:23.203045 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.203015    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pktv\" (UniqueName: \"kubernetes.io/projected/12c1ad92-cf64-47f8-aff2-3f1a9aebab7d-kube-api-access-5pktv\") pod \"kuadrant-console-plugin-66b5774985-f6kfh\" (UID: \"12c1ad92-cf64-47f8-aff2-3f1a9aebab7d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"
Jul 27 19:46:23.697251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.697211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/12c1ad92-cf64-47f8-aff2-3f1a9aebab7d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-f6kfh\" (UID: \"12c1ad92-cf64-47f8-aff2-3f1a9aebab7d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"
Jul 27 19:46:23.700094 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.700070    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/12c1ad92-cf64-47f8-aff2-3f1a9aebab7d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-f6kfh\" (UID: \"12c1ad92-cf64-47f8-aff2-3f1a9aebab7d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"
Jul 27 19:46:23.931816 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:23.931782    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"
Jul 27 19:46:24.068540 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:24.068514    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh"]
Jul 27 19:46:24.071028 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:46:24.071002    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12c1ad92_cf64_47f8_aff2_3f1a9aebab7d.slice/crio-aaddcdb5b6a4d00df4a2b35c44b633140a0f57b7bad8eebce3d451cd2a6e5f72 WatchSource:0}: Error finding container aaddcdb5b6a4d00df4a2b35c44b633140a0f57b7bad8eebce3d451cd2a6e5f72: Status 404 returned error can't find the container with id aaddcdb5b6a4d00df4a2b35c44b633140a0f57b7bad8eebce3d451cd2a6e5f72
Jul 27 19:46:24.765215 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:24.765139    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dgvzl" event={"ID":"f9938b9c-d76a-4c14-b8e1-95dad0295fea","Type":"ContainerStarted","Data":"c34ab6a99f3cf6bb006eceacb09d21841d1db4748f5de6760e48ff23c30fa2d1"}
Jul 27 19:46:24.765382 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:24.765259    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dgvzl"
Jul 27 19:46:24.766592 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:24.766553    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh" event={"ID":"12c1ad92-cf64-47f8-aff2-3f1a9aebab7d","Type":"ContainerStarted","Data":"aaddcdb5b6a4d00df4a2b35c44b633140a0f57b7bad8eebce3d451cd2a6e5f72"}
Jul 27 19:46:24.786758 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:24.786703    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dgvzl" podStartSLOduration=1.531697678 podStartE2EDuration="3.786685102s" totalImagesPullingTime="2.254987424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:46:24.784523387 +0000 UTC m=+399.336439809" watchObservedRunningTime="2026-07-27 19:46:24.786685102 +0000 UTC m=+399.338601520"
Jul 27 19:46:29.730367 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:29.730339    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:31.747149 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:31.747117    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:33.385654 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.385618    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"]
Jul 27 19:46:33.405732 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.405669    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"]
Jul 27 19:46:33.405732 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.405707    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:33.481312 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.481282    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab70ecc3-c617-4c6e-ad11-343852cd31cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-r7s9h\" (UID: \"ab70ecc3-c617-4c6e-ad11-343852cd31cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:33.481312 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.481323    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv88r\" (UniqueName: \"kubernetes.io/projected/ab70ecc3-c617-4c6e-ad11-343852cd31cc-kube-api-access-cv88r\") pod \"kuadrant-operator-controller-manager-55467d9f7-r7s9h\" (UID: \"ab70ecc3-c617-4c6e-ad11-343852cd31cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:33.581891 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.581847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab70ecc3-c617-4c6e-ad11-343852cd31cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-r7s9h\" (UID: \"ab70ecc3-c617-4c6e-ad11-343852cd31cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:33.582070 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.581904    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv88r\" (UniqueName: \"kubernetes.io/projected/ab70ecc3-c617-4c6e-ad11-343852cd31cc-kube-api-access-cv88r\") pod \"kuadrant-operator-controller-manager-55467d9f7-r7s9h\" (UID: \"ab70ecc3-c617-4c6e-ad11-343852cd31cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:33.582283 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.582260    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab70ecc3-c617-4c6e-ad11-343852cd31cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-r7s9h\" (UID: \"ab70ecc3-c617-4c6e-ad11-343852cd31cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:33.596908 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.596863    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv88r\" (UniqueName: \"kubernetes.io/projected/ab70ecc3-c617-4c6e-ad11-343852cd31cc-kube-api-access-cv88r\") pod \"kuadrant-operator-controller-manager-55467d9f7-r7s9h\" (UID: \"ab70ecc3-c617-4c6e-ad11-343852cd31cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:33.600225 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.600193    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"]
Jul 27 19:46:33.600654 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.600632    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:33.611159 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.611133    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"]
Jul 27 19:46:33.630498 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.630452    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"]
Jul 27 19:46:33.630724 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.630699    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd" podUID="ad24108b-7132-4567-8410-c0dc7815ccb9" containerName="manager" containerID="cri-o://c6ea806639af4bc4541ea2bcce49d8345d1f63ec75665295d1badeaaeb4a7edb" gracePeriod=2
Jul 27 19:46:33.651776 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.651707    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"]
Jul 27 19:46:33.662129 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.662096    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"]
Jul 27 19:46:33.662314 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:33.662277    2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ad24108b-7132-4567-8410-c0dc7815ccb9\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\": pods \"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:33.662724 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.662690    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad24108b-7132-4567-8410-c0dc7815ccb9" containerName="manager"
Jul 27 19:46:33.662724 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.662713    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad24108b-7132-4567-8410-c0dc7815ccb9" containerName="manager"
Jul 27 19:46:33.662864 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.662813    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad24108b-7132-4567-8410-c0dc7815ccb9" containerName="manager"
Jul 27 19:46:33.667908 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.667887    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"
Jul 27 19:46:33.669151 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.669093    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"]
Jul 27 19:46:33.669311 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.669280    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5" podUID="207e141f-84b7-4144-87b6-745374014cc8" containerName="manager" containerID="cri-o://f58bc781bdd815c4092c251785dad0eab1956866ee21055104ffc1c9c5f9ac48" gracePeriod=2
Jul 27 19:46:33.670089 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:33.670052    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="ad24108b-7132-4567-8410-c0dc7815ccb9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:33.692899 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.692281    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"]
Jul 27 19:46:33.694724 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.694627    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"]
Jul 27 19:46:33.716731 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.716705    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hx725"]
Jul 27 19:46:33.717206 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.717186    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="207e141f-84b7-4144-87b6-745374014cc8" containerName="manager"
Jul 27 19:46:33.717206 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.717210    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="207e141f-84b7-4144-87b6-745374014cc8" containerName="manager"
Jul 27 19:46:33.717412 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.717312    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="207e141f-84b7-4144-87b6-745374014cc8" containerName="manager"
Jul 27 19:46:33.724674 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.724648    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hx725"
Jul 27 19:46:33.739027 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:33.738987    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="ad24108b-7132-4567-8410-c0dc7815ccb9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:33.741663 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.741638    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hx725"]
Jul 27 19:46:33.760638 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.760564    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8hvxh"
Jul 27 19:46:33.763121 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:33.762940    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nz2g5\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="207e141f-84b7-4144-87b6-745374014cc8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:33.764883 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:33.764855    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="ad24108b-7132-4567-8410-c0dc7815ccb9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:33.766684 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:33.766659    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nz2g5\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="207e141f-84b7-4144-87b6-745374014cc8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:33.783899 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.783868    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bw6\" (UniqueName: \"kubernetes.io/projected/94196cc7-c5ac-468b-ba91-bde84894d140-kube-api-access-f7bw6\") pod \"kuadrant-operator-controller-manager-86878c769f-gwswd\" (UID: \"94196cc7-c5ac-468b-ba91-bde84894d140\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"
Jul 27 19:46:33.784022 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.783937    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlpzc\" (UniqueName: \"kubernetes.io/projected/6fa4c9e5-3210-49bb-a40a-f14b21d5af13-kube-api-access-mlpzc\") pod \"limitador-operator-controller-manager-859d7b78bd-hx725\" (UID: \"6fa4c9e5-3210-49bb-a40a-f14b21d5af13\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hx725"
Jul 27 19:46:33.784022 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.783968    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94196cc7-c5ac-468b-ba91-bde84894d140-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gwswd\" (UID: \"94196cc7-c5ac-468b-ba91-bde84894d140\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"
Jul 27 19:46:33.885457 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.885411    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bw6\" (UniqueName: \"kubernetes.io/projected/94196cc7-c5ac-468b-ba91-bde84894d140-kube-api-access-f7bw6\") pod \"kuadrant-operator-controller-manager-86878c769f-gwswd\" (UID: \"94196cc7-c5ac-468b-ba91-bde84894d140\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"
Jul 27 19:46:33.885662 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.885533    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlpzc\" (UniqueName: \"kubernetes.io/projected/6fa4c9e5-3210-49bb-a40a-f14b21d5af13-kube-api-access-mlpzc\") pod \"limitador-operator-controller-manager-859d7b78bd-hx725\" (UID: \"6fa4c9e5-3210-49bb-a40a-f14b21d5af13\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hx725"
Jul 27 19:46:33.885662 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.885572    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94196cc7-c5ac-468b-ba91-bde84894d140-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gwswd\" (UID: \"94196cc7-c5ac-468b-ba91-bde84894d140\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"
Jul 27 19:46:33.885984 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.885963    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94196cc7-c5ac-468b-ba91-bde84894d140-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gwswd\" (UID: \"94196cc7-c5ac-468b-ba91-bde84894d140\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"
Jul 27 19:46:33.899055 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.898989    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bw6\" (UniqueName: \"kubernetes.io/projected/94196cc7-c5ac-468b-ba91-bde84894d140-kube-api-access-f7bw6\") pod \"kuadrant-operator-controller-manager-86878c769f-gwswd\" (UID: \"94196cc7-c5ac-468b-ba91-bde84894d140\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"
Jul 27 19:46:33.901761 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:33.901739    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlpzc\" (UniqueName: \"kubernetes.io/projected/6fa4c9e5-3210-49bb-a40a-f14b21d5af13-kube-api-access-mlpzc\") pod \"limitador-operator-controller-manager-859d7b78bd-hx725\" (UID: \"6fa4c9e5-3210-49bb-a40a-f14b21d5af13\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hx725"
Jul 27 19:46:34.054609 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:34.054580    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"
Jul 27 19:46:34.064045 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:34.064018    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hx725"
Jul 27 19:46:35.774297 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:35.774262    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dgvzl"
Jul 27 19:46:45.610528 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.610498    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hx725"]
Jul 27 19:46:45.612977 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:46:45.612940    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa4c9e5_3210_49bb_a40a_f14b21d5af13.slice/crio-3b2f3b9801f5a0f7b4c722f9902691e72716bf6125ded0fd4630f5ea517956ef WatchSource:0}: Error finding container 3b2f3b9801f5a0f7b4c722f9902691e72716bf6125ded0fd4630f5ea517956ef: Status 404 returned error can't find the container with id 3b2f3b9801f5a0f7b4c722f9902691e72716bf6125ded0fd4630f5ea517956ef
Jul 27 19:46:45.624879 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.624858    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:45.626394 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:45.626363    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nz2g5\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="207e141f-84b7-4144-87b6-745374014cc8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:45.632989 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.632966    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:45.634749 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:45.634667    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="ad24108b-7132-4567-8410-c0dc7815ccb9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:45.636079 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:45.636047    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nz2g5\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="207e141f-84b7-4144-87b6-745374014cc8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:45.657612 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.657588    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"]
Jul 27 19:46:45.672649 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:46:45.672623    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94196cc7_c5ac_468b_ba91_bde84894d140.slice/crio-84d072d25137e5b968ea5e468867013d35b19ea2667a7e7e1ddc0e1944675baf WatchSource:0}: Error finding container 84d072d25137e5b968ea5e468867013d35b19ea2667a7e7e1ddc0e1944675baf: Status 404 returned error can't find the container with id 84d072d25137e5b968ea5e468867013d35b19ea2667a7e7e1ddc0e1944675baf
Jul 27 19:46:45.706267 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.706240    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzmvj\" (UniqueName: \"kubernetes.io/projected/ad24108b-7132-4567-8410-c0dc7815ccb9-kube-api-access-lzmvj\") pod \"ad24108b-7132-4567-8410-c0dc7815ccb9\" (UID: \"ad24108b-7132-4567-8410-c0dc7815ccb9\") "
Jul 27 19:46:45.706372 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.706348    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5zjv\" (UniqueName: \"kubernetes.io/projected/207e141f-84b7-4144-87b6-745374014cc8-kube-api-access-c5zjv\") pod \"207e141f-84b7-4144-87b6-745374014cc8\" (UID: \"207e141f-84b7-4144-87b6-745374014cc8\") "
Jul 27 19:46:45.706413 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.706400    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad24108b-7132-4567-8410-c0dc7815ccb9-extensions-socket-volume\") pod \"ad24108b-7132-4567-8410-c0dc7815ccb9\" (UID: \"ad24108b-7132-4567-8410-c0dc7815ccb9\") "
Jul 27 19:46:45.706915 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.706836    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad24108b-7132-4567-8410-c0dc7815ccb9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ad24108b-7132-4567-8410-c0dc7815ccb9" (UID: "ad24108b-7132-4567-8410-c0dc7815ccb9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:46:45.708333 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.708305    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad24108b-7132-4567-8410-c0dc7815ccb9-kube-api-access-lzmvj" (OuterVolumeSpecName: "kube-api-access-lzmvj") pod "ad24108b-7132-4567-8410-c0dc7815ccb9" (UID: "ad24108b-7132-4567-8410-c0dc7815ccb9"). InnerVolumeSpecName "kube-api-access-lzmvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:46:45.708407 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.708368    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207e141f-84b7-4144-87b6-745374014cc8-kube-api-access-c5zjv" (OuterVolumeSpecName: "kube-api-access-c5zjv") pod "207e141f-84b7-4144-87b6-745374014cc8" (UID: "207e141f-84b7-4144-87b6-745374014cc8"). InnerVolumeSpecName "kube-api-access-c5zjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:46:45.807897 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.807866    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5zjv\" (UniqueName: \"kubernetes.io/projected/207e141f-84b7-4144-87b6-745374014cc8-kube-api-access-c5zjv\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:46:45.807897 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.807892    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad24108b-7132-4567-8410-c0dc7815ccb9-extensions-socket-volume\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:46:45.807897 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.807903    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzmvj\" (UniqueName: \"kubernetes.io/projected/ad24108b-7132-4567-8410-c0dc7815ccb9-kube-api-access-lzmvj\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:46:45.881212 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.881173    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hx725" event={"ID":"6fa4c9e5-3210-49bb-a40a-f14b21d5af13","Type":"ContainerStarted","Data":"006a114f8a2c08234bdc392601fb696fbadbcd990cbcd67e5fe56dd63df3f543"}
Jul 27 19:46:45.881395 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.881382    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hx725"
Jul 27 19:46:45.881463 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.881437    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hx725" event={"ID":"6fa4c9e5-3210-49bb-a40a-f14b21d5af13","Type":"ContainerStarted","Data":"3b2f3b9801f5a0f7b4c722f9902691e72716bf6125ded0fd4630f5ea517956ef"}
Jul 27 19:46:45.882935 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:45.882905    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="ad24108b-7132-4567-8410-c0dc7815ccb9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:45.883075 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.882932    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh" event={"ID":"12c1ad92-cf64-47f8-aff2-3f1a9aebab7d","Type":"ContainerStarted","Data":"17b9e811336bcbaf3e4fffab6a79904886d843c0fdcafbe5ef87176154fbeccb"}
Jul 27 19:46:45.884759 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.884734    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd" event={"ID":"94196cc7-c5ac-468b-ba91-bde84894d140","Type":"ContainerStarted","Data":"e77227d293cf4ab59ca750a69695b5baaae8181517c3b5bf295b842a193e9b5d"}
Jul 27 19:46:45.884861 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.884781    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd" event={"ID":"94196cc7-c5ac-468b-ba91-bde84894d140","Type":"ContainerStarted","Data":"84d072d25137e5b968ea5e468867013d35b19ea2667a7e7e1ddc0e1944675baf"}
Jul 27 19:46:45.884861 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.884849    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"
Jul 27 19:46:45.886088 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.886066    2562 generic.go:356] "Generic (PLEG): container finished" podID="ad24108b-7132-4567-8410-c0dc7815ccb9" containerID="c6ea806639af4bc4541ea2bcce49d8345d1f63ec75665295d1badeaaeb4a7edb" exitCode=0
Jul 27 19:46:45.886175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.886093    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:45.886175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.886151    2562 scope.go:117] "RemoveContainer" containerID="c6ea806639af4bc4541ea2bcce49d8345d1f63ec75665295d1badeaaeb4a7edb"
Jul 27 19:46:45.887634 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.887612    2562 generic.go:356] "Generic (PLEG): container finished" podID="207e141f-84b7-4144-87b6-745374014cc8" containerID="f58bc781bdd815c4092c251785dad0eab1956866ee21055104ffc1c9c5f9ac48" exitCode=0
Jul 27 19:46:45.887900 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.887669    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:45.897212 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.897196    2562 scope.go:117] "RemoveContainer" containerID="c6ea806639af4bc4541ea2bcce49d8345d1f63ec75665295d1badeaaeb4a7edb"
Jul 27 19:46:45.897605 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:45.897583    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ea806639af4bc4541ea2bcce49d8345d1f63ec75665295d1badeaaeb4a7edb\": container with ID starting with c6ea806639af4bc4541ea2bcce49d8345d1f63ec75665295d1badeaaeb4a7edb not found: ID does not exist" containerID="c6ea806639af4bc4541ea2bcce49d8345d1f63ec75665295d1badeaaeb4a7edb"
Jul 27 19:46:45.897666 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.897617    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ea806639af4bc4541ea2bcce49d8345d1f63ec75665295d1badeaaeb4a7edb"} err="failed to get container status \"c6ea806639af4bc4541ea2bcce49d8345d1f63ec75665295d1badeaaeb4a7edb\": rpc error: code = NotFound desc = could not find container \"c6ea806639af4bc4541ea2bcce49d8345d1f63ec75665295d1badeaaeb4a7edb\": container with ID starting with c6ea806639af4bc4541ea2bcce49d8345d1f63ec75665295d1badeaaeb4a7edb not found: ID does not exist"
Jul 27 19:46:45.897666 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.897640    2562 scope.go:117] "RemoveContainer" containerID="f58bc781bdd815c4092c251785dad0eab1956866ee21055104ffc1c9c5f9ac48"
Jul 27 19:46:45.901066 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.901020    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hx725" podStartSLOduration=12.901003167 podStartE2EDuration="12.901003167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:46:45.898712733 +0000 UTC m=+420.450629152" watchObservedRunningTime="2026-07-27 19:46:45.901003167 +0000 UTC m=+420.452919567"
Jul 27 19:46:45.901587 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:45.901162    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nz2g5\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="207e141f-84b7-4144-87b6-745374014cc8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:45.902730 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:45.902702    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="ad24108b-7132-4567-8410-c0dc7815ccb9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:45.908083 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.908054    2562 scope.go:117] "RemoveContainer" containerID="f58bc781bdd815c4092c251785dad0eab1956866ee21055104ffc1c9c5f9ac48"
Jul 27 19:46:45.908356 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:45.908338    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f58bc781bdd815c4092c251785dad0eab1956866ee21055104ffc1c9c5f9ac48\": container with ID starting with f58bc781bdd815c4092c251785dad0eab1956866ee21055104ffc1c9c5f9ac48 not found: ID does not exist" containerID="f58bc781bdd815c4092c251785dad0eab1956866ee21055104ffc1c9c5f9ac48"
Jul 27 19:46:45.908414 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.908364    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f58bc781bdd815c4092c251785dad0eab1956866ee21055104ffc1c9c5f9ac48"} err="failed to get container status \"f58bc781bdd815c4092c251785dad0eab1956866ee21055104ffc1c9c5f9ac48\": rpc error: code = NotFound desc = could not find container \"f58bc781bdd815c4092c251785dad0eab1956866ee21055104ffc1c9c5f9ac48\": container with ID starting with f58bc781bdd815c4092c251785dad0eab1956866ee21055104ffc1c9c5f9ac48 not found: ID does not exist"
Jul 27 19:46:45.918383 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.918335    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6kfh" podStartSLOduration=2.478818658 podStartE2EDuration="23.918318593s" totalImagesPullingTime="21.439499935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:46:45.918118309 +0000 UTC m=+420.470034728" watchObservedRunningTime="2026-07-27 19:46:45.918318593 +0000 UTC m=+420.470235011"
Jul 27 19:46:45.919703 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:45.919674    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nz2g5\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="207e141f-84b7-4144-87b6-745374014cc8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:45.939198 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.939153    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd" podStartSLOduration=12.939137753 podStartE2EDuration="12.939137753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:46:45.937300968 +0000 UTC m=+420.489217387" watchObservedRunningTime="2026-07-27 19:46:45.939137753 +0000 UTC m=+420.491054172"
Jul 27 19:46:45.988748 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:45.988708    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nz2g5\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="207e141f-84b7-4144-87b6-745374014cc8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nz2g5"
Jul 27 19:46:45.990543 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:45.990515    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8hfgd\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="ad24108b-7132-4567-8410-c0dc7815ccb9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8hfgd"
Jul 27 19:46:45.991445 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.991412    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="207e141f-84b7-4144-87b6-745374014cc8" path="/var/lib/kubelet/pods/207e141f-84b7-4144-87b6-745374014cc8/volumes"
Jul 27 19:46:45.991830 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:45.991816    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad24108b-7132-4567-8410-c0dc7815ccb9" path="/var/lib/kubelet/pods/ad24108b-7132-4567-8410-c0dc7815ccb9/volumes"
Jul 27 19:46:47.066732 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:47.066694    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice/crio-f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice/crio-13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8\": RecentStats: unable to find data in memory cache]"
Jul 27 19:46:47.965492 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:47.965440    2562 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-r7s9h_kuadrant-system_ab70ecc3-c617-4c6e-ad11-343852cd31cc_0(20698832b0e8cd6844441605d0eda09331e38a180a3e86d9b1f4872f2065f292): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-r7s9h to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Jul 27 19:46:47.965572 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:47.965518    2562 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-r7s9h_kuadrant-system_ab70ecc3-c617-4c6e-ad11-343852cd31cc_0(20698832b0e8cd6844441605d0eda09331e38a180a3e86d9b1f4872f2065f292): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-r7s9h to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:48.565421 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:48.565381    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice/crio-13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice/crio-f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67\": RecentStats: unable to find data in memory cache]"
Jul 27 19:46:48.901534 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:48.901502    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:48.904507 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:48.904454    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-r7s9h\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="ab70ecc3-c617-4c6e-ad11-343852cd31cc" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:48.990454 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:48.990419    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:48.992447 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:48.992409    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-r7s9h\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="ab70ecc3-c617-4c6e-ad11-343852cd31cc" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:49.136011 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:49.135976    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab70ecc3-c617-4c6e-ad11-343852cd31cc-extensions-socket-volume\") pod \"ab70ecc3-c617-4c6e-ad11-343852cd31cc\" (UID: \"ab70ecc3-c617-4c6e-ad11-343852cd31cc\") "
Jul 27 19:46:49.136191 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:49.136076    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv88r\" (UniqueName: \"kubernetes.io/projected/ab70ecc3-c617-4c6e-ad11-343852cd31cc-kube-api-access-cv88r\") pod \"ab70ecc3-c617-4c6e-ad11-343852cd31cc\" (UID: \"ab70ecc3-c617-4c6e-ad11-343852cd31cc\") "
Jul 27 19:46:49.136256 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:49.136217    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab70ecc3-c617-4c6e-ad11-343852cd31cc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ab70ecc3-c617-4c6e-ad11-343852cd31cc" (UID: "ab70ecc3-c617-4c6e-ad11-343852cd31cc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:46:49.138334 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:49.138302    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab70ecc3-c617-4c6e-ad11-343852cd31cc-kube-api-access-cv88r" (OuterVolumeSpecName: "kube-api-access-cv88r") pod "ab70ecc3-c617-4c6e-ad11-343852cd31cc" (UID: "ab70ecc3-c617-4c6e-ad11-343852cd31cc"). InnerVolumeSpecName "kube-api-access-cv88r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:46:49.237385 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:49.237307    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab70ecc3-c617-4c6e-ad11-343852cd31cc-extensions-socket-volume\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:46:49.237385 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:49.237338    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv88r\" (UniqueName: \"kubernetes.io/projected/ab70ecc3-c617-4c6e-ad11-343852cd31cc-kube-api-access-cv88r\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:46:49.905996 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:49.905966    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:49.907858 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:49.907833    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-r7s9h\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="ab70ecc3-c617-4c6e-ad11-343852cd31cc" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:49.913076 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:49.913049    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-r7s9h\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="ab70ecc3-c617-4c6e-ad11-343852cd31cc" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-r7s9h"
Jul 27 19:46:49.990903 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:49.990873    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab70ecc3-c617-4c6e-ad11-343852cd31cc" path="/var/lib/kubelet/pods/ab70ecc3-c617-4c6e-ad11-343852cd31cc/volumes"
Jul 27 19:46:56.894035 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:56.893999    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hx725"
Jul 27 19:46:56.894628 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:46:56.894091    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"
Jul 27 19:46:57.109191 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:46:57.109152    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice/crio-13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice/crio-f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67\": RecentStats: unable to find data in memory cache]"
Jul 27 19:47:01.528507 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:47:01.528449    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice/crio-13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice/crio-f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 19:47:01.528507 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:47:01.528471    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice/crio-13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice/crio-f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 19:47:03.613641 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:47:03.613601    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice/crio-f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice/crio-13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 19:47:07.118604 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:47:07.118519    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice/crio-f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice/crio-13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8\": RecentStats: unable to find data in memory cache]"
Jul 27 19:47:13.253646 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.253617    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"]
Jul 27 19:47:13.257176 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.257157    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.260204 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.260184    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ppvh2\""
Jul 27 19:47:13.270498 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.270456    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"]
Jul 27 19:47:13.349870 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.349835    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.350062 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.349882    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.350062 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.349928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.350062 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.349955    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9c4g\" (UniqueName: \"kubernetes.io/projected/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-kube-api-access-c9c4g\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.350062 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.349987    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.350277 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.350061    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.350277 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.350119    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.350277 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.350148    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.350277 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.350178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.450844 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.450809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.450844 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.450844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.451100 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.450862    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.451100 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.450885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.451100 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.450923    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.451100 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.450955    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.451100 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.450981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9c4g\" (UniqueName: \"kubernetes.io/projected/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-kube-api-access-c9c4g\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.451100 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.451027    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.451100 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.451071    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.451433 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.451307    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.451433 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.451414    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.452063 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.452034    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.452200 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.452160    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.452401 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.452378    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.453568 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.453543    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.453765 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.453747    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.458922 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.458902    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9c4g\" (UniqueName: \"kubernetes.io/projected/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-kube-api-access-c9c4g\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.458999 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.458970    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861d7f38-16d8-41ed-bf20-d15a7bcc6cbf-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vbkhz\" (UID: \"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.567523 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.567409    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:13.711775 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.711667    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"]
Jul 27 19:47:13.714232 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:47:13.714206    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861d7f38_16d8_41ed_bf20_d15a7bcc6cbf.slice/crio-b7ac6a5ac171b82a80f6134d03544e40f9096688601af273f0f208fdc6f1088d WatchSource:0}: Error finding container b7ac6a5ac171b82a80f6134d03544e40f9096688601af273f0f208fdc6f1088d: Status 404 returned error can't find the container with id b7ac6a5ac171b82a80f6134d03544e40f9096688601af273f0f208fdc6f1088d
Jul 27 19:47:13.716384 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.716349    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 19:47:13.716468 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.716443    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 19:47:13.716536 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:13.716471    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 19:47:14.018636 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:14.018599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz" event={"ID":"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf","Type":"ContainerStarted","Data":"5e28b094012b5a852ad1d35f1c188180de3146b8d2cf9f2dd97ec22e5b4688cb"}
Jul 27 19:47:14.018809 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:14.018657    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz" event={"ID":"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf","Type":"ContainerStarted","Data":"b7ac6a5ac171b82a80f6134d03544e40f9096688601af273f0f208fdc6f1088d"}
Jul 27 19:47:14.041033 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:14.040981    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz" podStartSLOduration=1.040966007 podStartE2EDuration="1.040966007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:47:14.039625323 +0000 UTC m=+448.591541745" watchObservedRunningTime="2026-07-27 19:47:14.040966007 +0000 UTC m=+448.592882425"
Jul 27 19:47:14.568428 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:14.568387    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:14.569922 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:14.569894    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vbkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 27 19:47:14.570059 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:14.569951    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz" podUID="861d7f38-16d8-41ed-bf20-d15a7bcc6cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 27 19:47:15.157018 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:15.156980    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"]
Jul 27 19:47:15.189804 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:15.189769    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"]
Jul 27 19:47:15.189966 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:15.189882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"
Jul 27 19:47:15.192161 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:15.192138    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 19:47:15.245950 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:15.245921    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"]
Jul 27 19:47:15.268959 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:15.268930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/446a2cf4-60e4-497a-be1f-150d18710c76-config-file\") pod \"limitador-limitador-64c94fd64f-pxb8l\" (UID: \"446a2cf4-60e4-497a-be1f-150d18710c76\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"
Jul 27 19:47:15.269127 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:15.269007    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7gjk\" (UniqueName: \"kubernetes.io/projected/446a2cf4-60e4-497a-be1f-150d18710c76-kube-api-access-j7gjk\") pod \"limitador-limitador-64c94fd64f-pxb8l\" (UID: \"446a2cf4-60e4-497a-be1f-150d18710c76\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"
Jul 27 19:47:15.369553 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:15.369520    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7gjk\" (UniqueName: \"kubernetes.io/projected/446a2cf4-60e4-497a-be1f-150d18710c76-kube-api-access-j7gjk\") pod \"limitador-limitador-64c94fd64f-pxb8l\" (UID: \"446a2cf4-60e4-497a-be1f-150d18710c76\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"
Jul 27 19:47:15.369714 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:15.369579    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/446a2cf4-60e4-497a-be1f-150d18710c76-config-file\") pod \"limitador-limitador-64c94fd64f-pxb8l\" (UID: \"446a2cf4-60e4-497a-be1f-150d18710c76\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"
Jul 27 19:47:15.370192 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:15.370172    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/446a2cf4-60e4-497a-be1f-150d18710c76-config-file\") pod \"limitador-limitador-64c94fd64f-pxb8l\" (UID: \"446a2cf4-60e4-497a-be1f-150d18710c76\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"
Jul 27 19:47:15.377394 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:15.377372    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7gjk\" (UniqueName: \"kubernetes.io/projected/446a2cf4-60e4-497a-be1f-150d18710c76-kube-api-access-j7gjk\") pod \"limitador-limitador-64c94fd64f-pxb8l\" (UID: \"446a2cf4-60e4-497a-be1f-150d18710c76\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"
Jul 27 19:47:15.501712 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:15.501618    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"
Jul 27 19:47:15.571782 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:15.571753    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:15.641826 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:15.641796    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"]
Jul 27 19:47:15.645569 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:47:15.645543    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446a2cf4_60e4_497a_be1f_150d18710c76.slice/crio-77c2c7c74d43f93e90ab3bea212ec09d0144436c2b526854f29075751925e8cd WatchSource:0}: Error finding container 77c2c7c74d43f93e90ab3bea212ec09d0144436c2b526854f29075751925e8cd: Status 404 returned error can't find the container with id 77c2c7c74d43f93e90ab3bea212ec09d0144436c2b526854f29075751925e8cd
Jul 27 19:47:16.028954 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:16.028915    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l" event={"ID":"446a2cf4-60e4-497a-be1f-150d18710c76","Type":"ContainerStarted","Data":"77c2c7c74d43f93e90ab3bea212ec09d0144436c2b526854f29075751925e8cd"}
Jul 27 19:47:16.029180 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:16.029160    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:16.030116 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:16.030101    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:47:17.170766 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:47:17.170727    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice/crio-f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice/crio-13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 19:47:18.576762 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:47:18.576724    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice/crio-13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice/crio-f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 19:47:19.046379 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:19.046350    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l" event={"ID":"446a2cf4-60e4-497a-be1f-150d18710c76","Type":"ContainerStarted","Data":"0eda983f477b80d446fe75b2fd5396f8daa80cefa2f1298825509f558b361efd"}
Jul 27 19:47:19.046568 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:19.046505    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"
Jul 27 19:47:19.068153 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:19.068103    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l" podStartSLOduration=0.920462517 podStartE2EDuration="4.068085996s" totalImagesPullingTime="3.147623479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:47:19.066133583 +0000 UTC m=+453.618050001" watchObservedRunningTime="2026-07-27 19:47:19.068085996 +0000 UTC m=+453.620002425"
Jul 27 19:47:27.215228 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:47:27.215190    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice/crio-13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice/crio-f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 19:47:30.053074 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:30.053045    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"
Jul 27 19:47:30.218750 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:30.218714    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"]
Jul 27 19:47:30.218919 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:30.218904    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l" podUID="446a2cf4-60e4-497a-be1f-150d18710c76" containerName="limitador" containerID="cri-o://0eda983f477b80d446fe75b2fd5396f8daa80cefa2f1298825509f558b361efd" gracePeriod=30
Jul 27 19:47:30.760009 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:30.759985    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"
Jul 27 19:47:30.908795 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:30.908756    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/446a2cf4-60e4-497a-be1f-150d18710c76-config-file\") pod \"446a2cf4-60e4-497a-be1f-150d18710c76\" (UID: \"446a2cf4-60e4-497a-be1f-150d18710c76\") "
Jul 27 19:47:30.908966 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:30.908885    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7gjk\" (UniqueName: \"kubernetes.io/projected/446a2cf4-60e4-497a-be1f-150d18710c76-kube-api-access-j7gjk\") pod \"446a2cf4-60e4-497a-be1f-150d18710c76\" (UID: \"446a2cf4-60e4-497a-be1f-150d18710c76\") "
Jul 27 19:47:30.909174 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:30.909155    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446a2cf4-60e4-497a-be1f-150d18710c76-config-file" (OuterVolumeSpecName: "config-file") pod "446a2cf4-60e4-497a-be1f-150d18710c76" (UID: "446a2cf4-60e4-497a-be1f-150d18710c76"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:47:30.911031 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:30.911010    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446a2cf4-60e4-497a-be1f-150d18710c76-kube-api-access-j7gjk" (OuterVolumeSpecName: "kube-api-access-j7gjk") pod "446a2cf4-60e4-497a-be1f-150d18710c76" (UID: "446a2cf4-60e4-497a-be1f-150d18710c76"). InnerVolumeSpecName "kube-api-access-j7gjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:47:31.009853 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:31.009810    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7gjk\" (UniqueName: \"kubernetes.io/projected/446a2cf4-60e4-497a-be1f-150d18710c76-kube-api-access-j7gjk\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:47:31.009853 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:31.009844    2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/446a2cf4-60e4-497a-be1f-150d18710c76-config-file\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:47:31.104745 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:31.104706    2562 generic.go:356] "Generic (PLEG): container finished" podID="446a2cf4-60e4-497a-be1f-150d18710c76" containerID="0eda983f477b80d446fe75b2fd5396f8daa80cefa2f1298825509f558b361efd" exitCode=0
Jul 27 19:47:31.104745 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:31.104745    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"
Jul 27 19:47:31.105274 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:31.104773    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l" event={"ID":"446a2cf4-60e4-497a-be1f-150d18710c76","Type":"ContainerDied","Data":"0eda983f477b80d446fe75b2fd5396f8daa80cefa2f1298825509f558b361efd"}
Jul 27 19:47:31.105274 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:31.104824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pxb8l" event={"ID":"446a2cf4-60e4-497a-be1f-150d18710c76","Type":"ContainerDied","Data":"77c2c7c74d43f93e90ab3bea212ec09d0144436c2b526854f29075751925e8cd"}
Jul 27 19:47:31.105274 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:31.104852    2562 scope.go:117] "RemoveContainer" containerID="0eda983f477b80d446fe75b2fd5396f8daa80cefa2f1298825509f558b361efd"
Jul 27 19:47:31.114015 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:31.113998    2562 scope.go:117] "RemoveContainer" containerID="0eda983f477b80d446fe75b2fd5396f8daa80cefa2f1298825509f558b361efd"
Jul 27 19:47:31.114301 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:47:31.114283    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eda983f477b80d446fe75b2fd5396f8daa80cefa2f1298825509f558b361efd\": container with ID starting with 0eda983f477b80d446fe75b2fd5396f8daa80cefa2f1298825509f558b361efd not found: ID does not exist" containerID="0eda983f477b80d446fe75b2fd5396f8daa80cefa2f1298825509f558b361efd"
Jul 27 19:47:31.114357 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:31.114310    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eda983f477b80d446fe75b2fd5396f8daa80cefa2f1298825509f558b361efd"} err="failed to get container status \"0eda983f477b80d446fe75b2fd5396f8daa80cefa2f1298825509f558b361efd\": rpc error: code = NotFound desc = could not find container \"0eda983f477b80d446fe75b2fd5396f8daa80cefa2f1298825509f558b361efd\": container with ID starting with 0eda983f477b80d446fe75b2fd5396f8daa80cefa2f1298825509f558b361efd not found: ID does not exist"
Jul 27 19:47:31.125435 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:31.125408    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"]
Jul 27 19:47:31.127077 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:31.127056    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pxb8l"]
Jul 27 19:47:31.990603 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:31.990570    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="446a2cf4-60e4-497a-be1f-150d18710c76" path="/var/lib/kubelet/pods/446a2cf4-60e4-497a-be1f-150d18710c76/volumes"
Jul 27 19:47:33.618449 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:47:33.618404    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice/crio-13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice/crio-f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 19:47:36.642461 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.642427    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk"]
Jul 27 19:47:36.642949 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.642813    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="446a2cf4-60e4-497a-be1f-150d18710c76" containerName="limitador"
Jul 27 19:47:36.642949 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.642825    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="446a2cf4-60e4-497a-be1f-150d18710c76" containerName="limitador"
Jul 27 19:47:36.642949 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.642901    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="446a2cf4-60e4-497a-be1f-150d18710c76" containerName="limitador"
Jul 27 19:47:36.647165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.647144    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk"
Jul 27 19:47:36.649717 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.649690    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 19:47:36.649824 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.649734    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 19:47:36.649889 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.649701    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 27 19:47:36.650297 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.650280    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-cc6j9\""
Jul 27 19:47:36.657445 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.657423    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk"]
Jul 27 19:47:36.762058 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.762009    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/72b8cb4d-eae2-4786-853a-de7f919e38d7-data\") pod \"postgres-69fb4594cf-w6fbk\" (UID: \"72b8cb4d-eae2-4786-853a-de7f919e38d7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk"
Jul 27 19:47:36.762058 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.762079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4xqn\" (UniqueName: \"kubernetes.io/projected/72b8cb4d-eae2-4786-853a-de7f919e38d7-kube-api-access-r4xqn\") pod \"postgres-69fb4594cf-w6fbk\" (UID: \"72b8cb4d-eae2-4786-853a-de7f919e38d7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk"
Jul 27 19:47:36.863200 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.863163    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4xqn\" (UniqueName: \"kubernetes.io/projected/72b8cb4d-eae2-4786-853a-de7f919e38d7-kube-api-access-r4xqn\") pod \"postgres-69fb4594cf-w6fbk\" (UID: \"72b8cb4d-eae2-4786-853a-de7f919e38d7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk"
Jul 27 19:47:36.863374 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.863271    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/72b8cb4d-eae2-4786-853a-de7f919e38d7-data\") pod \"postgres-69fb4594cf-w6fbk\" (UID: \"72b8cb4d-eae2-4786-853a-de7f919e38d7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk"
Jul 27 19:47:36.863648 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.863632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/72b8cb4d-eae2-4786-853a-de7f919e38d7-data\") pod \"postgres-69fb4594cf-w6fbk\" (UID: \"72b8cb4d-eae2-4786-853a-de7f919e38d7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk"
Jul 27 19:47:36.873986 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.873962    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4xqn\" (UniqueName: \"kubernetes.io/projected/72b8cb4d-eae2-4786-853a-de7f919e38d7-kube-api-access-r4xqn\") pod \"postgres-69fb4594cf-w6fbk\" (UID: \"72b8cb4d-eae2-4786-853a-de7f919e38d7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk"
Jul 27 19:47:36.959693 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:36.959599    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk"
Jul 27 19:47:37.096501 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:37.096461    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk"]
Jul 27 19:47:37.134707 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:37.134673    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk" event={"ID":"72b8cb4d-eae2-4786-853a-de7f919e38d7","Type":"ContainerStarted","Data":"5e15c7230bbe37ccb8e7550e6d17171568433947502cb30fc3b2e85e7eb4e720"}
Jul 27 19:47:37.224873 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:47:37.224785    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice/crio-13d3905f2511a6bf7be00847ea6d685a90a9fece690508cbdf92adb5b04416f8\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice/crio-f4ddb9d4df776904519b6c6290fd524dca34820e317248ddd1effb2c90528d67\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24108b_7132_4567_8410_c0dc7815ccb9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e141f_84b7_4144_87b6_745374014cc8.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 19:47:42.165317 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:42.165283    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk" event={"ID":"72b8cb4d-eae2-4786-853a-de7f919e38d7","Type":"ContainerStarted","Data":"560513db08a2eb331cf7503cf14841c5c66b4de80c0d9cd2d605f01919d04325"}
Jul 27 19:47:42.165725 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:42.165408    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk"
Jul 27 19:47:42.184535 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:42.184456    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk" podStartSLOduration=1.269664291 podStartE2EDuration="6.184435935s" totalImagesPullingTime="4.914771644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:47:42.181003334 +0000 UTC m=+476.732919752" watchObservedRunningTime="2026-07-27 19:47:42.184435935 +0000 UTC m=+476.736352355"
Jul 27 19:47:48.199737 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:48.199710    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-w6fbk"
Jul 27 19:47:55.138067 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.138034    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-dbbc69fcc-6j5lh"]
Jul 27 19:47:55.165930 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.165900    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-dbbc69fcc-6j5lh"]
Jul 27 19:47:55.166071 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.166010    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dbbc69fcc-6j5lh"
Jul 27 19:47:55.168053 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.168029    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 19:47:55.168244 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.168207    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 19:47:55.168468 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.168451    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7mzv6\""
Jul 27 19:47:55.221113 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.221082    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a9cf84c5-b13d-4492-9d99-b3270efe72e2-webhook-cert\") pod \"maas-controller-dbbc69fcc-6j5lh\" (UID: \"a9cf84c5-b13d-4492-9d99-b3270efe72e2\") " pod="opendatahub/maas-controller-dbbc69fcc-6j5lh"
Jul 27 19:47:55.221272 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.221134    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrwcd\" (UniqueName: \"kubernetes.io/projected/a9cf84c5-b13d-4492-9d99-b3270efe72e2-kube-api-access-hrwcd\") pod \"maas-controller-dbbc69fcc-6j5lh\" (UID: \"a9cf84c5-b13d-4492-9d99-b3270efe72e2\") " pod="opendatahub/maas-controller-dbbc69fcc-6j5lh"
Jul 27 19:47:55.306818 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.306783    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c8b889c85-2fzbp"]
Jul 27 19:47:55.309975 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.309959    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8b889c85-2fzbp"
Jul 27 19:47:55.321508 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.321457    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8b889c85-2fzbp"]
Jul 27 19:47:55.321833 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.321811    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90de8a92-a8fe-41f0-a4be-349e9043f262-webhook-cert\") pod \"maas-controller-c8b889c85-2fzbp\" (UID: \"90de8a92-a8fe-41f0-a4be-349e9043f262\") " pod="opendatahub/maas-controller-c8b889c85-2fzbp"
Jul 27 19:47:55.321908 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.321894    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a9cf84c5-b13d-4492-9d99-b3270efe72e2-webhook-cert\") pod \"maas-controller-dbbc69fcc-6j5lh\" (UID: \"a9cf84c5-b13d-4492-9d99-b3270efe72e2\") " pod="opendatahub/maas-controller-dbbc69fcc-6j5lh"
Jul 27 19:47:55.321953 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.321918    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggfgz\" (UniqueName: \"kubernetes.io/projected/90de8a92-a8fe-41f0-a4be-349e9043f262-kube-api-access-ggfgz\") pod \"maas-controller-c8b889c85-2fzbp\" (UID: \"90de8a92-a8fe-41f0-a4be-349e9043f262\") " pod="opendatahub/maas-controller-c8b889c85-2fzbp"
Jul 27 19:47:55.321953 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.321941    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrwcd\" (UniqueName: \"kubernetes.io/projected/a9cf84c5-b13d-4492-9d99-b3270efe72e2-kube-api-access-hrwcd\") pod \"maas-controller-dbbc69fcc-6j5lh\" (UID: \"a9cf84c5-b13d-4492-9d99-b3270efe72e2\") " pod="opendatahub/maas-controller-dbbc69fcc-6j5lh"
Jul 27 19:47:55.324433 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.324402    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a9cf84c5-b13d-4492-9d99-b3270efe72e2-webhook-cert\") pod \"maas-controller-dbbc69fcc-6j5lh\" (UID: \"a9cf84c5-b13d-4492-9d99-b3270efe72e2\") " pod="opendatahub/maas-controller-dbbc69fcc-6j5lh"
Jul 27 19:47:55.335650 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.335631    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrwcd\" (UniqueName: \"kubernetes.io/projected/a9cf84c5-b13d-4492-9d99-b3270efe72e2-kube-api-access-hrwcd\") pod \"maas-controller-dbbc69fcc-6j5lh\" (UID: \"a9cf84c5-b13d-4492-9d99-b3270efe72e2\") " pod="opendatahub/maas-controller-dbbc69fcc-6j5lh"
Jul 27 19:47:55.423251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.423171    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90de8a92-a8fe-41f0-a4be-349e9043f262-webhook-cert\") pod \"maas-controller-c8b889c85-2fzbp\" (UID: \"90de8a92-a8fe-41f0-a4be-349e9043f262\") " pod="opendatahub/maas-controller-c8b889c85-2fzbp"
Jul 27 19:47:55.423251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.423230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggfgz\" (UniqueName: \"kubernetes.io/projected/90de8a92-a8fe-41f0-a4be-349e9043f262-kube-api-access-ggfgz\") pod \"maas-controller-c8b889c85-2fzbp\" (UID: \"90de8a92-a8fe-41f0-a4be-349e9043f262\") " pod="opendatahub/maas-controller-c8b889c85-2fzbp"
Jul 27 19:47:55.425754 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.425725    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90de8a92-a8fe-41f0-a4be-349e9043f262-webhook-cert\") pod \"maas-controller-c8b889c85-2fzbp\" (UID: \"90de8a92-a8fe-41f0-a4be-349e9043f262\") " pod="opendatahub/maas-controller-c8b889c85-2fzbp"
Jul 27 19:47:55.431829 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.431804    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggfgz\" (UniqueName: \"kubernetes.io/projected/90de8a92-a8fe-41f0-a4be-349e9043f262-kube-api-access-ggfgz\") pod \"maas-controller-c8b889c85-2fzbp\" (UID: \"90de8a92-a8fe-41f0-a4be-349e9043f262\") " pod="opendatahub/maas-controller-c8b889c85-2fzbp"
Jul 27 19:47:55.477728 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.477698    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dbbc69fcc-6j5lh"
Jul 27 19:47:55.613220 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.613198    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-dbbc69fcc-6j5lh"]
Jul 27 19:47:55.615824 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:47:55.615788    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cf84c5_b13d_4492_9d99_b3270efe72e2.slice/crio-6f1b290e59c1fd76843b2e5e6b31d4eab9b5397b72c5f800aba8b67fbd896b42 WatchSource:0}: Error finding container 6f1b290e59c1fd76843b2e5e6b31d4eab9b5397b72c5f800aba8b67fbd896b42: Status 404 returned error can't find the container with id 6f1b290e59c1fd76843b2e5e6b31d4eab9b5397b72c5f800aba8b67fbd896b42
Jul 27 19:47:55.620682 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.620663    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8b889c85-2fzbp"
Jul 27 19:47:55.754126 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:55.754046    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8b889c85-2fzbp"]
Jul 27 19:47:55.756409 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:47:55.756376    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90de8a92_a8fe_41f0_a4be_349e9043f262.slice/crio-1017e79d485dfdbd5e5771badbeabb41dd69c2c18c00e0a932553a0600d289a6 WatchSource:0}: Error finding container 1017e79d485dfdbd5e5771badbeabb41dd69c2c18c00e0a932553a0600d289a6: Status 404 returned error can't find the container with id 1017e79d485dfdbd5e5771badbeabb41dd69c2c18c00e0a932553a0600d289a6
Jul 27 19:47:56.233686 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:56.233644    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8b889c85-2fzbp" event={"ID":"90de8a92-a8fe-41f0-a4be-349e9043f262","Type":"ContainerStarted","Data":"1017e79d485dfdbd5e5771badbeabb41dd69c2c18c00e0a932553a0600d289a6"}
Jul 27 19:47:56.235845 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:56.235813    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dbbc69fcc-6j5lh" event={"ID":"a9cf84c5-b13d-4492-9d99-b3270efe72e2","Type":"ContainerStarted","Data":"6f1b290e59c1fd76843b2e5e6b31d4eab9b5397b72c5f800aba8b67fbd896b42"}
Jul 27 19:47:58.248263 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:58.248237    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8b889c85-2fzbp" event={"ID":"90de8a92-a8fe-41f0-a4be-349e9043f262","Type":"ContainerStarted","Data":"1f9fc0c594de8225e36ef57a90eabc51723612c52bb4d554c61d98852f55816c"}
Jul 27 19:47:58.248647 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:58.248356    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c8b889c85-2fzbp"
Jul 27 19:47:58.249996 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:58.249972    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dbbc69fcc-6j5lh" event={"ID":"a9cf84c5-b13d-4492-9d99-b3270efe72e2","Type":"ContainerStarted","Data":"dc8bfea4d08366038acea98d4437e2ae17cae2083ef5e9f912214dd687aca5da"}
Jul 27 19:47:58.266381 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:58.266330    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c8b889c85-2fzbp" podStartSLOduration=0.846040389 podStartE2EDuration="3.266314772s" totalImagesPullingTime="2.420274383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:47:58.264774021 +0000 UTC m=+492.816690439" watchObservedRunningTime="2026-07-27 19:47:58.266314772 +0000 UTC m=+492.818231190"
Jul 27 19:47:58.281303 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:58.281257    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-dbbc69fcc-6j5lh" podStartSLOduration=0.726617456 podStartE2EDuration="3.281242348s" totalImagesPullingTime="2.554624892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:47:58.280291472 +0000 UTC m=+492.832207891" watchObservedRunningTime="2026-07-27 19:47:58.281242348 +0000 UTC m=+492.833158766"
Jul 27 19:47:59.254773 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:47:59.254737    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-dbbc69fcc-6j5lh"
Jul 27 19:48:09.257554 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:09.257517    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c8b889c85-2fzbp"
Jul 27 19:48:09.307217 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:09.307184    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-dbbc69fcc-6j5lh"]
Jul 27 19:48:09.307428 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:09.307403    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-dbbc69fcc-6j5lh" podUID="a9cf84c5-b13d-4492-9d99-b3270efe72e2" containerName="manager" containerID="cri-o://dc8bfea4d08366038acea98d4437e2ae17cae2083ef5e9f912214dd687aca5da" gracePeriod=30
Jul 27 19:48:09.309421 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:09.309397    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-dbbc69fcc-6j5lh"
Jul 27 19:48:09.557683 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:09.557661    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dbbc69fcc-6j5lh"
Jul 27 19:48:09.655298 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:09.655264    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrwcd\" (UniqueName: \"kubernetes.io/projected/a9cf84c5-b13d-4492-9d99-b3270efe72e2-kube-api-access-hrwcd\") pod \"a9cf84c5-b13d-4492-9d99-b3270efe72e2\" (UID: \"a9cf84c5-b13d-4492-9d99-b3270efe72e2\") "
Jul 27 19:48:09.655467 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:09.655326    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a9cf84c5-b13d-4492-9d99-b3270efe72e2-webhook-cert\") pod \"a9cf84c5-b13d-4492-9d99-b3270efe72e2\" (UID: \"a9cf84c5-b13d-4492-9d99-b3270efe72e2\") "
Jul 27 19:48:09.657642 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:09.657613    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cf84c5-b13d-4492-9d99-b3270efe72e2-kube-api-access-hrwcd" (OuterVolumeSpecName: "kube-api-access-hrwcd") pod "a9cf84c5-b13d-4492-9d99-b3270efe72e2" (UID: "a9cf84c5-b13d-4492-9d99-b3270efe72e2"). InnerVolumeSpecName "kube-api-access-hrwcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:48:09.657748 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:09.657726    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cf84c5-b13d-4492-9d99-b3270efe72e2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a9cf84c5-b13d-4492-9d99-b3270efe72e2" (UID: "a9cf84c5-b13d-4492-9d99-b3270efe72e2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:48:09.756284 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:09.756250    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a9cf84c5-b13d-4492-9d99-b3270efe72e2-webhook-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:48:09.756284 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:09.756278    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrwcd\" (UniqueName: \"kubernetes.io/projected/a9cf84c5-b13d-4492-9d99-b3270efe72e2-kube-api-access-hrwcd\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:48:10.312743 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:10.312656    2562 generic.go:356] "Generic (PLEG): container finished" podID="a9cf84c5-b13d-4492-9d99-b3270efe72e2" containerID="dc8bfea4d08366038acea98d4437e2ae17cae2083ef5e9f912214dd687aca5da" exitCode=0
Jul 27 19:48:10.312743 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:10.312701    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dbbc69fcc-6j5lh"
Jul 27 19:48:10.313187 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:10.312755    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dbbc69fcc-6j5lh" event={"ID":"a9cf84c5-b13d-4492-9d99-b3270efe72e2","Type":"ContainerDied","Data":"dc8bfea4d08366038acea98d4437e2ae17cae2083ef5e9f912214dd687aca5da"}
Jul 27 19:48:10.313187 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:10.312805    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dbbc69fcc-6j5lh" event={"ID":"a9cf84c5-b13d-4492-9d99-b3270efe72e2","Type":"ContainerDied","Data":"6f1b290e59c1fd76843b2e5e6b31d4eab9b5397b72c5f800aba8b67fbd896b42"}
Jul 27 19:48:10.313187 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:10.312827    2562 scope.go:117] "RemoveContainer" containerID="dc8bfea4d08366038acea98d4437e2ae17cae2083ef5e9f912214dd687aca5da"
Jul 27 19:48:10.321105 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:10.321090    2562 scope.go:117] "RemoveContainer" containerID="dc8bfea4d08366038acea98d4437e2ae17cae2083ef5e9f912214dd687aca5da"
Jul 27 19:48:10.321360 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:48:10.321343    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc8bfea4d08366038acea98d4437e2ae17cae2083ef5e9f912214dd687aca5da\": container with ID starting with dc8bfea4d08366038acea98d4437e2ae17cae2083ef5e9f912214dd687aca5da not found: ID does not exist" containerID="dc8bfea4d08366038acea98d4437e2ae17cae2083ef5e9f912214dd687aca5da"
Jul 27 19:48:10.321415 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:10.321370    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc8bfea4d08366038acea98d4437e2ae17cae2083ef5e9f912214dd687aca5da"} err="failed to get container status \"dc8bfea4d08366038acea98d4437e2ae17cae2083ef5e9f912214dd687aca5da\": rpc error: code = NotFound desc = could not find container \"dc8bfea4d08366038acea98d4437e2ae17cae2083ef5e9f912214dd687aca5da\": container with ID starting with dc8bfea4d08366038acea98d4437e2ae17cae2083ef5e9f912214dd687aca5da not found: ID does not exist"
Jul 27 19:48:10.329383 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:10.329362    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-dbbc69fcc-6j5lh"]
Jul 27 19:48:10.332990 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:10.332972    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-dbbc69fcc-6j5lh"]
Jul 27 19:48:11.991389 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:11.991360    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9cf84c5-b13d-4492-9d99-b3270efe72e2" path="/var/lib/kubelet/pods/a9cf84c5-b13d-4492-9d99-b3270efe72e2/volumes"
Jul 27 19:48:30.829103 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:30.829062    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"]
Jul 27 19:48:30.829802 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:30.829662    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cf84c5-b13d-4492-9d99-b3270efe72e2" containerName="manager"
Jul 27 19:48:30.829802 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:30.829683    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cf84c5-b13d-4492-9d99-b3270efe72e2" containerName="manager"
Jul 27 19:48:30.829973 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:30.829812    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9cf84c5-b13d-4492-9d99-b3270efe72e2" containerName="manager"
Jul 27 19:48:30.834441 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:30.834421    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"
Jul 27 19:48:30.836966 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:30.836944    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-j4rqh\""
Jul 27 19:48:30.837067 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:30.836977    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 27 19:48:30.847442 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:30.847421    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"]
Jul 27 19:48:30.941767 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:30.941733    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60115a3c-980a-4762-a08d-40532cc0c63c-maas-api-tls\") pod \"maas-api-77b4b4d988-h85l4\" (UID: \"60115a3c-980a-4762-a08d-40532cc0c63c\") " pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"
Jul 27 19:48:30.941958 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:30.941792    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60115a3c-980a-4762-a08d-40532cc0c63c-openshift-service-ca\") pod \"maas-api-77b4b4d988-h85l4\" (UID: \"60115a3c-980a-4762-a08d-40532cc0c63c\") " pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"
Jul 27 19:48:30.941958 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:30.941905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfc7c\" (UniqueName: \"kubernetes.io/projected/60115a3c-980a-4762-a08d-40532cc0c63c-kube-api-access-xfc7c\") pod \"maas-api-77b4b4d988-h85l4\" (UID: \"60115a3c-980a-4762-a08d-40532cc0c63c\") " pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"
Jul 27 19:48:31.042748 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.042719    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60115a3c-980a-4762-a08d-40532cc0c63c-maas-api-tls\") pod \"maas-api-77b4b4d988-h85l4\" (UID: \"60115a3c-980a-4762-a08d-40532cc0c63c\") " pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"
Jul 27 19:48:31.042907 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.042760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60115a3c-980a-4762-a08d-40532cc0c63c-openshift-service-ca\") pod \"maas-api-77b4b4d988-h85l4\" (UID: \"60115a3c-980a-4762-a08d-40532cc0c63c\") " pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"
Jul 27 19:48:31.042907 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.042796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfc7c\" (UniqueName: \"kubernetes.io/projected/60115a3c-980a-4762-a08d-40532cc0c63c-kube-api-access-xfc7c\") pod \"maas-api-77b4b4d988-h85l4\" (UID: \"60115a3c-980a-4762-a08d-40532cc0c63c\") " pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"
Jul 27 19:48:31.043536 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.043517    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60115a3c-980a-4762-a08d-40532cc0c63c-openshift-service-ca\") pod \"maas-api-77b4b4d988-h85l4\" (UID: \"60115a3c-980a-4762-a08d-40532cc0c63c\") " pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"
Jul 27 19:48:31.045730 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.045704    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60115a3c-980a-4762-a08d-40532cc0c63c-maas-api-tls\") pod \"maas-api-77b4b4d988-h85l4\" (UID: \"60115a3c-980a-4762-a08d-40532cc0c63c\") " pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"
Jul 27 19:48:31.050852 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.050831    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfc7c\" (UniqueName: \"kubernetes.io/projected/60115a3c-980a-4762-a08d-40532cc0c63c-kube-api-access-xfc7c\") pod \"maas-api-77b4b4d988-h85l4\" (UID: \"60115a3c-980a-4762-a08d-40532cc0c63c\") " pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"
Jul 27 19:48:31.144498 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.144439    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"
Jul 27 19:48:31.289755 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.289728    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"]
Jul 27 19:48:31.292246 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:48:31.292215    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60115a3c_980a_4762_a08d_40532cc0c63c.slice/crio-b856eedc91696db2876bb58d7ace709749bab126c93ada087e62e7fd0dafc834 WatchSource:0}: Error finding container b856eedc91696db2876bb58d7ace709749bab126c93ada087e62e7fd0dafc834: Status 404 returned error can't find the container with id b856eedc91696db2876bb58d7ace709749bab126c93ada087e62e7fd0dafc834
Jul 27 19:48:31.362180 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.362152    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-wrscr"]
Jul 27 19:48:31.366874 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.366854    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-wrscr"
Jul 27 19:48:31.369132 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.369112    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5sfvj\""
Jul 27 19:48:31.369223 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.369112    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 27 19:48:31.377109 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.377084    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-wrscr"]
Jul 27 19:48:31.417803 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.417732    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4" event={"ID":"60115a3c-980a-4762-a08d-40532cc0c63c","Type":"ContainerStarted","Data":"b856eedc91696db2876bb58d7ace709749bab126c93ada087e62e7fd0dafc834"}
Jul 27 19:48:31.422378 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.422354    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-952m8"]
Jul 27 19:48:31.425846 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.425827    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-952m8"
Jul 27 19:48:31.439513 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.439492    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-952m8"]
Jul 27 19:48:31.446428 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.446391    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bda14a-6ab1-406e-8754-305218a633ab-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-wrscr\" (UID: \"65bda14a-6ab1-406e-8754-305218a633ab\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wrscr"
Jul 27 19:48:31.446574 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.446438    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm5xg\" (UniqueName: \"kubernetes.io/projected/65bda14a-6ab1-406e-8754-305218a633ab-kube-api-access-cm5xg\") pod \"payload-processing-5cc6cd444b-wrscr\" (UID: \"65bda14a-6ab1-406e-8754-305218a633ab\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wrscr"
Jul 27 19:48:31.547116 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.547075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6da65f36-1bc2-44a6-b6f7-52cf19b524d6-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-952m8\" (UID: \"6da65f36-1bc2-44a6-b6f7-52cf19b524d6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-952m8"
Jul 27 19:48:31.547286 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.547135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bda14a-6ab1-406e-8754-305218a633ab-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-wrscr\" (UID: \"65bda14a-6ab1-406e-8754-305218a633ab\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wrscr"
Jul 27 19:48:31.547286 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.547207    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm5xg\" (UniqueName: \"kubernetes.io/projected/65bda14a-6ab1-406e-8754-305218a633ab-kube-api-access-cm5xg\") pod \"payload-processing-5cc6cd444b-wrscr\" (UID: \"65bda14a-6ab1-406e-8754-305218a633ab\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wrscr"
Jul 27 19:48:31.547286 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.547257    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csgpm\" (UniqueName: \"kubernetes.io/projected/6da65f36-1bc2-44a6-b6f7-52cf19b524d6-kube-api-access-csgpm\") pod \"payload-pre-processing-687dbb6cf6-952m8\" (UID: \"6da65f36-1bc2-44a6-b6f7-52cf19b524d6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-952m8"
Jul 27 19:48:31.548041 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.548017    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bda14a-6ab1-406e-8754-305218a633ab-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-wrscr\" (UID: \"65bda14a-6ab1-406e-8754-305218a633ab\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wrscr"
Jul 27 19:48:31.555896 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.555873    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm5xg\" (UniqueName: \"kubernetes.io/projected/65bda14a-6ab1-406e-8754-305218a633ab-kube-api-access-cm5xg\") pod \"payload-processing-5cc6cd444b-wrscr\" (UID: \"65bda14a-6ab1-406e-8754-305218a633ab\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wrscr"
Jul 27 19:48:31.648544 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.648508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6da65f36-1bc2-44a6-b6f7-52cf19b524d6-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-952m8\" (UID: \"6da65f36-1bc2-44a6-b6f7-52cf19b524d6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-952m8"
Jul 27 19:48:31.648694 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.648589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csgpm\" (UniqueName: \"kubernetes.io/projected/6da65f36-1bc2-44a6-b6f7-52cf19b524d6-kube-api-access-csgpm\") pod \"payload-pre-processing-687dbb6cf6-952m8\" (UID: \"6da65f36-1bc2-44a6-b6f7-52cf19b524d6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-952m8"
Jul 27 19:48:31.649169 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.649151    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6da65f36-1bc2-44a6-b6f7-52cf19b524d6-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-952m8\" (UID: \"6da65f36-1bc2-44a6-b6f7-52cf19b524d6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-952m8"
Jul 27 19:48:31.656094 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.656070    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csgpm\" (UniqueName: \"kubernetes.io/projected/6da65f36-1bc2-44a6-b6f7-52cf19b524d6-kube-api-access-csgpm\") pod \"payload-pre-processing-687dbb6cf6-952m8\" (UID: \"6da65f36-1bc2-44a6-b6f7-52cf19b524d6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-952m8"
Jul 27 19:48:31.678679 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.678608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-wrscr"
Jul 27 19:48:31.736876 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.736776    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-952m8"
Jul 27 19:48:31.830088 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.830060    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-wrscr"]
Jul 27 19:48:31.832395 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:48:31.832366    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bda14a_6ab1_406e_8754_305218a633ab.slice/crio-82bbfaa7bb1a28b0d6e8b9af861302f12d25ed80a662684feec7d7403c3fc812 WatchSource:0}: Error finding container 82bbfaa7bb1a28b0d6e8b9af861302f12d25ed80a662684feec7d7403c3fc812: Status 404 returned error can't find the container with id 82bbfaa7bb1a28b0d6e8b9af861302f12d25ed80a662684feec7d7403c3fc812
Jul 27 19:48:31.893349 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:31.893311    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-952m8"]
Jul 27 19:48:31.897180 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:48:31.897151    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da65f36_1bc2_44a6_b6f7_52cf19b524d6.slice/crio-557fab209af27c1721dbdcbdc345371862210972da4b75bc31332914ab2253c8 WatchSource:0}: Error finding container 557fab209af27c1721dbdcbdc345371862210972da4b75bc31332914ab2253c8: Status 404 returned error can't find the container with id 557fab209af27c1721dbdcbdc345371862210972da4b75bc31332914ab2253c8
Jul 27 19:48:32.424727 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:32.424676    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-wrscr" event={"ID":"65bda14a-6ab1-406e-8754-305218a633ab","Type":"ContainerStarted","Data":"82bbfaa7bb1a28b0d6e8b9af861302f12d25ed80a662684feec7d7403c3fc812"}
Jul 27 19:48:32.426128 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:32.426083    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-952m8" event={"ID":"6da65f36-1bc2-44a6-b6f7-52cf19b524d6","Type":"ContainerStarted","Data":"557fab209af27c1721dbdcbdc345371862210972da4b75bc31332914ab2253c8"}
Jul 27 19:48:34.443171 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:34.443128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4" event={"ID":"60115a3c-980a-4762-a08d-40532cc0c63c","Type":"ContainerStarted","Data":"fe39cc5f9b1219957f588cd1e65a27e6c94176b9d15aa841b1baebcc9d9cc81f"}
Jul 27 19:48:34.443584 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:34.443273    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"
Jul 27 19:48:34.461932 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:34.461872    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4" podStartSLOduration=1.69521107 podStartE2EDuration="4.461856611s" totalImagesPullingTime="2.766645541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:48:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:48:34.460079447 +0000 UTC m=+529.011995867" watchObservedRunningTime="2026-07-27 19:48:34.461856611 +0000 UTC m=+529.013773029"
Jul 27 19:48:36.454960 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:36.454912    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-wrscr" event={"ID":"65bda14a-6ab1-406e-8754-305218a633ab","Type":"ContainerStarted","Data":"84c357d3c086a909789aae2942937b4016a62860a422920d52a86db0a6372223"}
Jul 27 19:48:36.455522 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:36.455123    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-wrscr"
Jul 27 19:48:36.456491 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:36.456459    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-952m8" event={"ID":"6da65f36-1bc2-44a6-b6f7-52cf19b524d6","Type":"ContainerStarted","Data":"12b3185402425b7e4f1447fe48611547f9a77ec0d59509b3d214f563e71f0654"}
Jul 27 19:48:36.456657 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:36.456605    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-952m8"
Jul 27 19:48:36.475631 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:36.475576    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-wrscr" podStartSLOduration=1.7668459460000001 podStartE2EDuration="5.475559438s" totalImagesPullingTime="3.708713492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:48:36.473392666 +0000 UTC m=+531.025309084" watchObservedRunningTime="2026-07-27 19:48:36.475559438 +0000 UTC m=+531.027475857"
Jul 27 19:48:36.493815 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:36.493769    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-952m8" podStartSLOduration=1.843709383 podStartE2EDuration="5.493759372s" totalImagesPullingTime="3.650049989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:48:36.490568938 +0000 UTC m=+531.042485357" watchObservedRunningTime="2026-07-27 19:48:36.493759372 +0000 UTC m=+531.045675789"
Jul 27 19:48:39.472949 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:39.472918    2562 generic.go:356] "Generic (PLEG): container finished" podID="861d7f38-16d8-41ed-bf20-d15a7bcc6cbf" containerID="5e28b094012b5a852ad1d35f1c188180de3146b8d2cf9f2dd97ec22e5b4688cb" exitCode=0
Jul 27 19:48:39.473377 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:39.472998    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz" event={"ID":"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf","Type":"ContainerDied","Data":"5e28b094012b5a852ad1d35f1c188180de3146b8d2cf9f2dd97ec22e5b4688cb"}
Jul 27 19:48:39.473420 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:39.473384    2562 scope.go:117] "RemoveContainer" containerID="5e28b094012b5a852ad1d35f1c188180de3146b8d2cf9f2dd97ec22e5b4688cb"
Jul 27 19:48:39.474398 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:39.474373    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 19:48:39.474453 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:39.474438    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 19:48:39.474517 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:39.474469    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 19:48:40.454868 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:40.454839    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-77b4b4d988-h85l4"
Jul 27 19:48:40.486796 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:40.486762    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz" event={"ID":"861d7f38-16d8-41ed-bf20-d15a7bcc6cbf","Type":"ContainerStarted","Data":"8dcda4668bb57a0012cd503769884e88589fa7fb04d1d3e909f7a6ac4688c025"}
Jul 27 19:48:40.567693 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:40.567661    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:48:40.568272 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:40.568247    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vbkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 27 19:48:40.568380 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:40.568292    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz" podUID="861d7f38-16d8-41ed-bf20-d15a7bcc6cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 27 19:48:41.568764 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:41.568728    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vbkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 27 19:48:41.569148 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:41.568783    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz" podUID="861d7f38-16d8-41ed-bf20-d15a7bcc6cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 27 19:48:42.568235 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:42.568195    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vbkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 27 19:48:42.568418 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:42.568274    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz" podUID="861d7f38-16d8-41ed-bf20-d15a7bcc6cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 27 19:48:43.568595 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:43.568549    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vbkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 27 19:48:43.568986 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:43.568621    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz" podUID="861d7f38-16d8-41ed-bf20-d15a7bcc6cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 27 19:48:44.585990 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:44.585947    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:48:44.586407 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:44.586244    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:48:44.587273 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:44.587249    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vbkhz"
Jul 27 19:48:46.825175 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:46.825142    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"]
Jul 27 19:48:46.830859 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:46.830829    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:46.832955 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:46.832932    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 19:48:46.833071 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:46.832990    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pvx6j\""
Jul 27 19:48:46.833403 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:46.833388    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 19:48:46.833447 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:46.833406    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 27 19:48:46.839416 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:46.839394    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"]
Jul 27 19:48:46.942412 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:46.942373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:46.942614 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:46.942439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:46.942614 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:46.942465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:46.942614 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:46.942516    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j7pt\" (UniqueName: \"kubernetes.io/projected/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-kube-api-access-2j7pt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:46.942614 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:46.942593    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:46.942827 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:46.942618    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:47.043036 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.043002    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:47.043219 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.043080    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:47.043219 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.043101    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:47.043219 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.043126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j7pt\" (UniqueName: \"kubernetes.io/projected/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-kube-api-access-2j7pt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:47.043219 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.043158    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:47.043219 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.043171    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:47.043518 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.043434    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:47.043576 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.043562    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:47.043617 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.043565    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:47.045469 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.045450    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:47.045783 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.045766    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:47.051281 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.051259    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j7pt\" (UniqueName: \"kubernetes.io/projected/b9e9e33b-ea90-4d81-8285-20dc7e7bea67-kube-api-access-2j7pt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz\" (UID: \"b9e9e33b-ea90-4d81-8285-20dc7e7bea67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:47.143169 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.143135    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:47.287028 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.286941    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"]
Jul 27 19:48:47.290104 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:48:47.290070    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e9e33b_ea90_4d81_8285_20dc7e7bea67.slice/crio-ad5a21a190bb3f063e8456524ef11544b9b28fca15c2b3d022e66bdf7b7f7062 WatchSource:0}: Error finding container ad5a21a190bb3f063e8456524ef11544b9b28fca15c2b3d022e66bdf7b7f7062: Status 404 returned error can't find the container with id ad5a21a190bb3f063e8456524ef11544b9b28fca15c2b3d022e66bdf7b7f7062
Jul 27 19:48:47.463764 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.463670    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-wrscr"
Jul 27 19:48:47.463764 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.463737    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-952m8"
Jul 27 19:48:47.533159 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:47.533122    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz" event={"ID":"b9e9e33b-ea90-4d81-8285-20dc7e7bea67","Type":"ContainerStarted","Data":"ad5a21a190bb3f063e8456524ef11544b9b28fca15c2b3d022e66bdf7b7f7062"}
Jul 27 19:48:48.539590 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:48.539555    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz" event={"ID":"b9e9e33b-ea90-4d81-8285-20dc7e7bea67","Type":"ContainerStarted","Data":"25c29ba65f7a4ec77b98c6af81277da44ad1a24f6d3f5698b703d8a8cdee46c8"}
Jul 27 19:48:48.571505 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:48.571425    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz" podStartSLOduration=1.452174297 podStartE2EDuration="2.571410105s" totalImagesPullingTime="1.119235808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:48:48.570302404 +0000 UTC m=+543.122218822" watchObservedRunningTime="2026-07-27 19:48:48.571410105 +0000 UTC m=+543.123326523"
Jul 27 19:48:57.143733 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:57.143697    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:57.144189 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:57.143749    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:57.156118 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:57.156094    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:48:57.596286 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:48:57.596210    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz"
Jul 27 19:51:43.460468 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:43.460367    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c8b889c85-2fzbp"]
Jul 27 19:51:43.460979 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:43.460692    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c8b889c85-2fzbp" podUID="90de8a92-a8fe-41f0-a4be-349e9043f262" containerName="manager" containerID="cri-o://1f9fc0c594de8225e36ef57a90eabc51723612c52bb4d554c61d98852f55816c" gracePeriod=30
Jul 27 19:51:43.716960 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:43.716894    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8b889c85-2fzbp"
Jul 27 19:51:43.857990 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:43.857958    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggfgz\" (UniqueName: \"kubernetes.io/projected/90de8a92-a8fe-41f0-a4be-349e9043f262-kube-api-access-ggfgz\") pod \"90de8a92-a8fe-41f0-a4be-349e9043f262\" (UID: \"90de8a92-a8fe-41f0-a4be-349e9043f262\") "
Jul 27 19:51:43.858165 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:43.858060    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90de8a92-a8fe-41f0-a4be-349e9043f262-webhook-cert\") pod \"90de8a92-a8fe-41f0-a4be-349e9043f262\" (UID: \"90de8a92-a8fe-41f0-a4be-349e9043f262\") "
Jul 27 19:51:43.860323 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:43.860288    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90de8a92-a8fe-41f0-a4be-349e9043f262-kube-api-access-ggfgz" (OuterVolumeSpecName: "kube-api-access-ggfgz") pod "90de8a92-a8fe-41f0-a4be-349e9043f262" (UID: "90de8a92-a8fe-41f0-a4be-349e9043f262"). InnerVolumeSpecName "kube-api-access-ggfgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:51:43.860323 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:43.860304    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90de8a92-a8fe-41f0-a4be-349e9043f262-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "90de8a92-a8fe-41f0-a4be-349e9043f262" (UID: "90de8a92-a8fe-41f0-a4be-349e9043f262"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:51:43.959153 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:43.959119    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90de8a92-a8fe-41f0-a4be-349e9043f262-webhook-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:51:43.959153 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:43.959147    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggfgz\" (UniqueName: \"kubernetes.io/projected/90de8a92-a8fe-41f0-a4be-349e9043f262-kube-api-access-ggfgz\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:51:44.402009 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:44.401969    2562 generic.go:356] "Generic (PLEG): container finished" podID="90de8a92-a8fe-41f0-a4be-349e9043f262" containerID="1f9fc0c594de8225e36ef57a90eabc51723612c52bb4d554c61d98852f55816c" exitCode=0
Jul 27 19:51:44.402192 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:44.402017    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8b889c85-2fzbp"
Jul 27 19:51:44.402192 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:44.402057    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8b889c85-2fzbp" event={"ID":"90de8a92-a8fe-41f0-a4be-349e9043f262","Type":"ContainerDied","Data":"1f9fc0c594de8225e36ef57a90eabc51723612c52bb4d554c61d98852f55816c"}
Jul 27 19:51:44.402192 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:44.402114    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8b889c85-2fzbp" event={"ID":"90de8a92-a8fe-41f0-a4be-349e9043f262","Type":"ContainerDied","Data":"1017e79d485dfdbd5e5771badbeabb41dd69c2c18c00e0a932553a0600d289a6"}
Jul 27 19:51:44.402192 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:44.402136    2562 scope.go:117] "RemoveContainer" containerID="1f9fc0c594de8225e36ef57a90eabc51723612c52bb4d554c61d98852f55816c"
Jul 27 19:51:44.410679 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:44.410660    2562 scope.go:117] "RemoveContainer" containerID="1f9fc0c594de8225e36ef57a90eabc51723612c52bb4d554c61d98852f55816c"
Jul 27 19:51:44.410952 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:51:44.410931    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9fc0c594de8225e36ef57a90eabc51723612c52bb4d554c61d98852f55816c\": container with ID starting with 1f9fc0c594de8225e36ef57a90eabc51723612c52bb4d554c61d98852f55816c not found: ID does not exist" containerID="1f9fc0c594de8225e36ef57a90eabc51723612c52bb4d554c61d98852f55816c"
Jul 27 19:51:44.411006 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:44.410962    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9fc0c594de8225e36ef57a90eabc51723612c52bb4d554c61d98852f55816c"} err="failed to get container status \"1f9fc0c594de8225e36ef57a90eabc51723612c52bb4d554c61d98852f55816c\": rpc error: code = NotFound desc = could not find container \"1f9fc0c594de8225e36ef57a90eabc51723612c52bb4d554c61d98852f55816c\": container with ID starting with 1f9fc0c594de8225e36ef57a90eabc51723612c52bb4d554c61d98852f55816c not found: ID does not exist"
Jul 27 19:51:44.418740 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:44.418715    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c8b889c85-2fzbp"]
Jul 27 19:51:44.423714 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:44.423689    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c8b889c85-2fzbp"]
Jul 27 19:51:45.991948 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:51:45.991923    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90de8a92-a8fe-41f0-a4be-349e9043f262" path="/var/lib/kubelet/pods/90de8a92-a8fe-41f0-a4be-349e9043f262/volumes"
Jul 27 19:52:45.200438 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.200409    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c8b889c85-npjdq"]
Jul 27 19:52:45.200863 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.200834    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90de8a92-a8fe-41f0-a4be-349e9043f262" containerName="manager"
Jul 27 19:52:45.200863 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.200849    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="90de8a92-a8fe-41f0-a4be-349e9043f262" containerName="manager"
Jul 27 19:52:45.200935 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.200927    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="90de8a92-a8fe-41f0-a4be-349e9043f262" containerName="manager"
Jul 27 19:52:45.203725 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.203709    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8b889c85-npjdq"
Jul 27 19:52:45.205860 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.205836    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 19:52:45.206495 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.206455    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 19:52:45.206577 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.206456    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7mzv6\""
Jul 27 19:52:45.212741 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.212721    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8b889c85-npjdq"]
Jul 27 19:52:45.315770 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.315731    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qwx\" (UniqueName: \"kubernetes.io/projected/480eb7d8-2560-49e1-805e-8e0297e8bbfb-kube-api-access-z7qwx\") pod \"maas-controller-c8b889c85-npjdq\" (UID: \"480eb7d8-2560-49e1-805e-8e0297e8bbfb\") " pod="opendatahub/maas-controller-c8b889c85-npjdq"
Jul 27 19:52:45.315965 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.315797    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/480eb7d8-2560-49e1-805e-8e0297e8bbfb-webhook-cert\") pod \"maas-controller-c8b889c85-npjdq\" (UID: \"480eb7d8-2560-49e1-805e-8e0297e8bbfb\") " pod="opendatahub/maas-controller-c8b889c85-npjdq"
Jul 27 19:52:45.416322 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.416276    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qwx\" (UniqueName: \"kubernetes.io/projected/480eb7d8-2560-49e1-805e-8e0297e8bbfb-kube-api-access-z7qwx\") pod \"maas-controller-c8b889c85-npjdq\" (UID: \"480eb7d8-2560-49e1-805e-8e0297e8bbfb\") " pod="opendatahub/maas-controller-c8b889c85-npjdq"
Jul 27 19:52:45.416570 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.416358    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/480eb7d8-2560-49e1-805e-8e0297e8bbfb-webhook-cert\") pod \"maas-controller-c8b889c85-npjdq\" (UID: \"480eb7d8-2560-49e1-805e-8e0297e8bbfb\") " pod="opendatahub/maas-controller-c8b889c85-npjdq"
Jul 27 19:52:45.418965 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.418940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/480eb7d8-2560-49e1-805e-8e0297e8bbfb-webhook-cert\") pod \"maas-controller-c8b889c85-npjdq\" (UID: \"480eb7d8-2560-49e1-805e-8e0297e8bbfb\") " pod="opendatahub/maas-controller-c8b889c85-npjdq"
Jul 27 19:52:45.424620 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.424602    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qwx\" (UniqueName: \"kubernetes.io/projected/480eb7d8-2560-49e1-805e-8e0297e8bbfb-kube-api-access-z7qwx\") pod \"maas-controller-c8b889c85-npjdq\" (UID: \"480eb7d8-2560-49e1-805e-8e0297e8bbfb\") " pod="opendatahub/maas-controller-c8b889c85-npjdq"
Jul 27 19:52:45.515873 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.515779    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8b889c85-npjdq"
Jul 27 19:52:45.657429 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.657396    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8b889c85-npjdq"]
Jul 27 19:52:45.661617 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:52:45.661590    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480eb7d8_2560_49e1_805e_8e0297e8bbfb.slice/crio-dcda07650ddf0f425610b9957a7828c29cf1c9279da539d44162ebf5d450f9bf WatchSource:0}: Error finding container dcda07650ddf0f425610b9957a7828c29cf1c9279da539d44162ebf5d450f9bf: Status 404 returned error can't find the container with id dcda07650ddf0f425610b9957a7828c29cf1c9279da539d44162ebf5d450f9bf
Jul 27 19:52:45.663152 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.663136    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:52:45.708767 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:45.708731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8b889c85-npjdq" event={"ID":"480eb7d8-2560-49e1-805e-8e0297e8bbfb","Type":"ContainerStarted","Data":"dcda07650ddf0f425610b9957a7828c29cf1c9279da539d44162ebf5d450f9bf"}
Jul 27 19:52:46.113845 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:46.113823    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 19:52:46.715537 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:46.715497    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8b889c85-npjdq" event={"ID":"480eb7d8-2560-49e1-805e-8e0297e8bbfb","Type":"ContainerStarted","Data":"25c2b8c8402f1867a6fa824db0e6dbb62a4a98191355408548a3e18475093b82"}
Jul 27 19:52:46.716026 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:46.715628    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c8b889c85-npjdq"
Jul 27 19:52:46.735947 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:46.735902    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c8b889c85-npjdq" podStartSLOduration=1.287551243 podStartE2EDuration="1.735888037s" totalImagesPullingTime="448.336794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:52:46.734416444 +0000 UTC m=+781.286332862" watchObservedRunningTime="2026-07-27 19:52:46.735888037 +0000 UTC m=+781.287804454"
Jul 27 19:52:57.723507 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:52:57.723450    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c8b889c85-npjdq"
Jul 27 19:53:02.346272 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.346227    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c8b889c85-npjdq"]
Jul 27 19:53:02.346788 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.346454    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c8b889c85-npjdq" podUID="480eb7d8-2560-49e1-805e-8e0297e8bbfb" containerName="manager" containerID="cri-o://25c2b8c8402f1867a6fa824db0e6dbb62a4a98191355408548a3e18475093b82" gracePeriod=30
Jul 27 19:53:02.599011 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.598949    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8b889c85-npjdq"
Jul 27 19:53:02.677602 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.677570    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7qwx\" (UniqueName: \"kubernetes.io/projected/480eb7d8-2560-49e1-805e-8e0297e8bbfb-kube-api-access-z7qwx\") pod \"480eb7d8-2560-49e1-805e-8e0297e8bbfb\" (UID: \"480eb7d8-2560-49e1-805e-8e0297e8bbfb\") "
Jul 27 19:53:02.677809 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.677615    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/480eb7d8-2560-49e1-805e-8e0297e8bbfb-webhook-cert\") pod \"480eb7d8-2560-49e1-805e-8e0297e8bbfb\" (UID: \"480eb7d8-2560-49e1-805e-8e0297e8bbfb\") "
Jul 27 19:53:02.679827 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.679798    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/480eb7d8-2560-49e1-805e-8e0297e8bbfb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "480eb7d8-2560-49e1-805e-8e0297e8bbfb" (UID: "480eb7d8-2560-49e1-805e-8e0297e8bbfb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:53:02.679942 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.679801    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480eb7d8-2560-49e1-805e-8e0297e8bbfb-kube-api-access-z7qwx" (OuterVolumeSpecName: "kube-api-access-z7qwx") pod "480eb7d8-2560-49e1-805e-8e0297e8bbfb" (UID: "480eb7d8-2560-49e1-805e-8e0297e8bbfb"). InnerVolumeSpecName "kube-api-access-z7qwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:53:02.779094 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.779051    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7qwx\" (UniqueName: \"kubernetes.io/projected/480eb7d8-2560-49e1-805e-8e0297e8bbfb-kube-api-access-z7qwx\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:53:02.779094 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.779093    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/480eb7d8-2560-49e1-805e-8e0297e8bbfb-webhook-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:53:02.793366 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.793335    2562 generic.go:356] "Generic (PLEG): container finished" podID="480eb7d8-2560-49e1-805e-8e0297e8bbfb" containerID="25c2b8c8402f1867a6fa824db0e6dbb62a4a98191355408548a3e18475093b82" exitCode=0
Jul 27 19:53:02.793566 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.793381    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8b889c85-npjdq"
Jul 27 19:53:02.793566 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.793427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8b889c85-npjdq" event={"ID":"480eb7d8-2560-49e1-805e-8e0297e8bbfb","Type":"ContainerDied","Data":"25c2b8c8402f1867a6fa824db0e6dbb62a4a98191355408548a3e18475093b82"}
Jul 27 19:53:02.793566 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.793512    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8b889c85-npjdq" event={"ID":"480eb7d8-2560-49e1-805e-8e0297e8bbfb","Type":"ContainerDied","Data":"dcda07650ddf0f425610b9957a7828c29cf1c9279da539d44162ebf5d450f9bf"}
Jul 27 19:53:02.793566 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.793544    2562 scope.go:117] "RemoveContainer" containerID="25c2b8c8402f1867a6fa824db0e6dbb62a4a98191355408548a3e18475093b82"
Jul 27 19:53:02.802794 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.802779    2562 scope.go:117] "RemoveContainer" containerID="25c2b8c8402f1867a6fa824db0e6dbb62a4a98191355408548a3e18475093b82"
Jul 27 19:53:02.803033 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:53:02.803015    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c2b8c8402f1867a6fa824db0e6dbb62a4a98191355408548a3e18475093b82\": container with ID starting with 25c2b8c8402f1867a6fa824db0e6dbb62a4a98191355408548a3e18475093b82 not found: ID does not exist" containerID="25c2b8c8402f1867a6fa824db0e6dbb62a4a98191355408548a3e18475093b82"
Jul 27 19:53:02.803080 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.803041    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c2b8c8402f1867a6fa824db0e6dbb62a4a98191355408548a3e18475093b82"} err="failed to get container status \"25c2b8c8402f1867a6fa824db0e6dbb62a4a98191355408548a3e18475093b82\": rpc error: code = NotFound desc = could not find container \"25c2b8c8402f1867a6fa824db0e6dbb62a4a98191355408548a3e18475093b82\": container with ID starting with 25c2b8c8402f1867a6fa824db0e6dbb62a4a98191355408548a3e18475093b82 not found: ID does not exist"
Jul 27 19:53:02.815679 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.815656    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c8b889c85-npjdq"]
Jul 27 19:53:02.819891 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:02.819871    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c8b889c85-npjdq"]
Jul 27 19:53:03.644900 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.644863    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c8b889c85-k7rvz"]
Jul 27 19:53:03.645299 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.645286    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480eb7d8-2560-49e1-805e-8e0297e8bbfb" containerName="manager"
Jul 27 19:53:03.645299 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.645300    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="480eb7d8-2560-49e1-805e-8e0297e8bbfb" containerName="manager"
Jul 27 19:53:03.645371 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.645364    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="480eb7d8-2560-49e1-805e-8e0297e8bbfb" containerName="manager"
Jul 27 19:53:03.649825 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.649807    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8b889c85-k7rvz"
Jul 27 19:53:03.652868 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.652843    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 19:53:03.652868 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.652862    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7mzv6\""
Jul 27 19:53:03.653055 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.652845    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 19:53:03.660174 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.659522    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8b889c85-k7rvz"]
Jul 27 19:53:03.789204 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.789170    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5-webhook-cert\") pod \"maas-controller-c8b889c85-k7rvz\" (UID: \"ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5\") " pod="opendatahub/maas-controller-c8b889c85-k7rvz"
Jul 27 19:53:03.789380 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.789230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zfs8\" (UniqueName: \"kubernetes.io/projected/ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5-kube-api-access-9zfs8\") pod \"maas-controller-c8b889c85-k7rvz\" (UID: \"ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5\") " pod="opendatahub/maas-controller-c8b889c85-k7rvz"
Jul 27 19:53:03.890660 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.890627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5-webhook-cert\") pod \"maas-controller-c8b889c85-k7rvz\" (UID: \"ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5\") " pod="opendatahub/maas-controller-c8b889c85-k7rvz"
Jul 27 19:53:03.890831 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.890687    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfs8\" (UniqueName: \"kubernetes.io/projected/ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5-kube-api-access-9zfs8\") pod \"maas-controller-c8b889c85-k7rvz\" (UID: \"ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5\") " pod="opendatahub/maas-controller-c8b889c85-k7rvz"
Jul 27 19:53:03.893213 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.893188    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5-webhook-cert\") pod \"maas-controller-c8b889c85-k7rvz\" (UID: \"ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5\") " pod="opendatahub/maas-controller-c8b889c85-k7rvz"
Jul 27 19:53:03.900339 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.900272    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zfs8\" (UniqueName: \"kubernetes.io/projected/ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5-kube-api-access-9zfs8\") pod \"maas-controller-c8b889c85-k7rvz\" (UID: \"ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5\") " pod="opendatahub/maas-controller-c8b889c85-k7rvz"
Jul 27 19:53:03.963738 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.963704    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8b889c85-k7rvz"
Jul 27 19:53:03.991251 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:03.991219    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="480eb7d8-2560-49e1-805e-8e0297e8bbfb" path="/var/lib/kubelet/pods/480eb7d8-2560-49e1-805e-8e0297e8bbfb/volumes"
Jul 27 19:53:04.103394 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:04.103365    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8b889c85-k7rvz"]
Jul 27 19:53:04.106038 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:53:04.106011    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7521f3_a8cb_49a4_a2bd_d3ce27080dc5.slice/crio-019079d0d3c57482b071a59074cdb607f859b74c02d676bf7ffe2b052afd0c31 WatchSource:0}: Error finding container 019079d0d3c57482b071a59074cdb607f859b74c02d676bf7ffe2b052afd0c31: Status 404 returned error can't find the container with id 019079d0d3c57482b071a59074cdb607f859b74c02d676bf7ffe2b052afd0c31
Jul 27 19:53:04.805769 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:04.805675    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8b889c85-k7rvz" event={"ID":"ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5","Type":"ContainerStarted","Data":"63f9567129440bb823b3d0d04720534efa5f4efab86f755a105e97990cbed35b"}
Jul 27 19:53:04.805769 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:04.805731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8b889c85-k7rvz" event={"ID":"ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5","Type":"ContainerStarted","Data":"019079d0d3c57482b071a59074cdb607f859b74c02d676bf7ffe2b052afd0c31"}
Jul 27 19:53:04.806180 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:04.805808    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c8b889c85-k7rvz"
Jul 27 19:53:04.825135 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:04.825082    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c8b889c85-k7rvz" podStartSLOduration=1.50115257 podStartE2EDuration="1.825066222s" totalImagesPullingTime="323.913652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:53:04.824056953 +0000 UTC m=+799.375973372" watchObservedRunningTime="2026-07-27 19:53:04.825066222 +0000 UTC m=+799.376982640"
Jul 27 19:53:15.813099 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:53:15.813015    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c8b889c85-k7rvz"
Jul 27 19:57:47.104678 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:47.104589    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"]
Jul 27 19:57:47.105167 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:47.104797    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd" podUID="94196cc7-c5ac-468b-ba91-bde84894d140" containerName="manager" containerID="cri-o://e77227d293cf4ab59ca750a69695b5baaae8181517c3b5bf295b842a193e9b5d" gracePeriod=10
Jul 27 19:57:49.563032 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:49.563007    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"
Jul 27 19:57:49.741903 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:49.741818    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94196cc7-c5ac-468b-ba91-bde84894d140-extensions-socket-volume\") pod \"94196cc7-c5ac-468b-ba91-bde84894d140\" (UID: \"94196cc7-c5ac-468b-ba91-bde84894d140\") "
Jul 27 19:57:49.741903 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:49.741874    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7bw6\" (UniqueName: \"kubernetes.io/projected/94196cc7-c5ac-468b-ba91-bde84894d140-kube-api-access-f7bw6\") pod \"94196cc7-c5ac-468b-ba91-bde84894d140\" (UID: \"94196cc7-c5ac-468b-ba91-bde84894d140\") "
Jul 27 19:57:49.742215 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:49.742194    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94196cc7-c5ac-468b-ba91-bde84894d140-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "94196cc7-c5ac-468b-ba91-bde84894d140" (UID: "94196cc7-c5ac-468b-ba91-bde84894d140"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:57:49.744267 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:49.744241    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94196cc7-c5ac-468b-ba91-bde84894d140-kube-api-access-f7bw6" (OuterVolumeSpecName: "kube-api-access-f7bw6") pod "94196cc7-c5ac-468b-ba91-bde84894d140" (UID: "94196cc7-c5ac-468b-ba91-bde84894d140"). InnerVolumeSpecName "kube-api-access-f7bw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:57:49.843282 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:49.843237    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94196cc7-c5ac-468b-ba91-bde84894d140-extensions-socket-volume\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:57:49.843282 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:49.843273    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7bw6\" (UniqueName: \"kubernetes.io/projected/94196cc7-c5ac-468b-ba91-bde84894d140-kube-api-access-f7bw6\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 19:57:50.219268 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:50.219228    2562 generic.go:356] "Generic (PLEG): container finished" podID="94196cc7-c5ac-468b-ba91-bde84894d140" containerID="e77227d293cf4ab59ca750a69695b5baaae8181517c3b5bf295b842a193e9b5d" exitCode=0
Jul 27 19:57:50.219464 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:50.219278    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"
Jul 27 19:57:50.219464 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:50.219311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd" event={"ID":"94196cc7-c5ac-468b-ba91-bde84894d140","Type":"ContainerDied","Data":"e77227d293cf4ab59ca750a69695b5baaae8181517c3b5bf295b842a193e9b5d"}
Jul 27 19:57:50.219464 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:50.219385    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd" event={"ID":"94196cc7-c5ac-468b-ba91-bde84894d140","Type":"ContainerDied","Data":"84d072d25137e5b968ea5e468867013d35b19ea2667a7e7e1ddc0e1944675baf"}
Jul 27 19:57:50.219464 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:50.219415    2562 scope.go:117] "RemoveContainer" containerID="e77227d293cf4ab59ca750a69695b5baaae8181517c3b5bf295b842a193e9b5d"
Jul 27 19:57:50.229187 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:50.229171    2562 scope.go:117] "RemoveContainer" containerID="e77227d293cf4ab59ca750a69695b5baaae8181517c3b5bf295b842a193e9b5d"
Jul 27 19:57:50.229466 ip-10-0-142-140 kubenswrapper[2562]: E0727 19:57:50.229443    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e77227d293cf4ab59ca750a69695b5baaae8181517c3b5bf295b842a193e9b5d\": container with ID starting with e77227d293cf4ab59ca750a69695b5baaae8181517c3b5bf295b842a193e9b5d not found: ID does not exist" containerID="e77227d293cf4ab59ca750a69695b5baaae8181517c3b5bf295b842a193e9b5d"
Jul 27 19:57:50.229584 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:50.229472    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e77227d293cf4ab59ca750a69695b5baaae8181517c3b5bf295b842a193e9b5d"} err="failed to get container status \"e77227d293cf4ab59ca750a69695b5baaae8181517c3b5bf295b842a193e9b5d\": rpc error: code = NotFound desc = could not find container \"e77227d293cf4ab59ca750a69695b5baaae8181517c3b5bf295b842a193e9b5d\": container with ID starting with e77227d293cf4ab59ca750a69695b5baaae8181517c3b5bf295b842a193e9b5d not found: ID does not exist"
Jul 27 19:57:50.236054 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:50.236033    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"]
Jul 27 19:57:50.240360 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:50.240341    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gwswd"]
Jul 27 19:57:51.991999 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:57:51.991968    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94196cc7-c5ac-468b-ba91-bde84894d140" path="/var/lib/kubelet/pods/94196cc7-c5ac-468b-ba91-bde84894d140/volumes"
Jul 27 19:58:48.229821 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.229783    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh"]
Jul 27 19:58:48.230221 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.230188    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94196cc7-c5ac-468b-ba91-bde84894d140" containerName="manager"
Jul 27 19:58:48.230221 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.230199    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="94196cc7-c5ac-468b-ba91-bde84894d140" containerName="manager"
Jul 27 19:58:48.230294 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.230255    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="94196cc7-c5ac-468b-ba91-bde84894d140" containerName="manager"
Jul 27 19:58:48.233704 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.233672    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh"
Jul 27 19:58:48.237527 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.237499    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-c5b2b\""
Jul 27 19:58:48.247978 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.247948    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh"]
Jul 27 19:58:48.255144 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.255120    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d7fce82-17ec-4d6c-ae75-c76cdafcd56b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dnrwh\" (UID: \"2d7fce82-17ec-4d6c-ae75-c76cdafcd56b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh"
Jul 27 19:58:48.255255 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.255151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnzqv\" (UniqueName: \"kubernetes.io/projected/2d7fce82-17ec-4d6c-ae75-c76cdafcd56b-kube-api-access-vnzqv\") pod \"kuadrant-operator-controller-manager-86878c769f-dnrwh\" (UID: \"2d7fce82-17ec-4d6c-ae75-c76cdafcd56b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh"
Jul 27 19:58:48.356570 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.356532    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d7fce82-17ec-4d6c-ae75-c76cdafcd56b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dnrwh\" (UID: \"2d7fce82-17ec-4d6c-ae75-c76cdafcd56b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh"
Jul 27 19:58:48.356570 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.356576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnzqv\" (UniqueName: \"kubernetes.io/projected/2d7fce82-17ec-4d6c-ae75-c76cdafcd56b-kube-api-access-vnzqv\") pod \"kuadrant-operator-controller-manager-86878c769f-dnrwh\" (UID: \"2d7fce82-17ec-4d6c-ae75-c76cdafcd56b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh"
Jul 27 19:58:48.356893 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.356870    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d7fce82-17ec-4d6c-ae75-c76cdafcd56b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dnrwh\" (UID: \"2d7fce82-17ec-4d6c-ae75-c76cdafcd56b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh"
Jul 27 19:58:48.365815 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.365785    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnzqv\" (UniqueName: \"kubernetes.io/projected/2d7fce82-17ec-4d6c-ae75-c76cdafcd56b-kube-api-access-vnzqv\") pod \"kuadrant-operator-controller-manager-86878c769f-dnrwh\" (UID: \"2d7fce82-17ec-4d6c-ae75-c76cdafcd56b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh"
Jul 27 19:58:48.544216 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.544105    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh"
Jul 27 19:58:48.706972 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:48.706946    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh"]
Jul 27 19:58:48.709546 ip-10-0-142-140 kubenswrapper[2562]: W0727 19:58:48.709512    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7fce82_17ec_4d6c_ae75_c76cdafcd56b.slice/crio-be44fe00bfc61d0c5697b417e231940e6739719bd9eaaa96d7b356fc4bf6717b WatchSource:0}: Error finding container be44fe00bfc61d0c5697b417e231940e6739719bd9eaaa96d7b356fc4bf6717b: Status 404 returned error can't find the container with id be44fe00bfc61d0c5697b417e231940e6739719bd9eaaa96d7b356fc4bf6717b
Jul 27 19:58:49.524855 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:49.524817    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh" event={"ID":"2d7fce82-17ec-4d6c-ae75-c76cdafcd56b","Type":"ContainerStarted","Data":"bc8758f6088d1c5fd0246541db4905d21da838403b9c5dc8f4327e5afe9e1637"}
Jul 27 19:58:49.525231 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:49.524977    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh"
Jul 27 19:58:49.525231 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:49.525017    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh" event={"ID":"2d7fce82-17ec-4d6c-ae75-c76cdafcd56b","Type":"ContainerStarted","Data":"be44fe00bfc61d0c5697b417e231940e6739719bd9eaaa96d7b356fc4bf6717b"}
Jul 27 19:58:49.548469 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:58:49.548416    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh" podStartSLOduration=1.548401568 podStartE2EDuration="1.548401568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:58:49.546075791 +0000 UTC m=+1144.097992210" watchObservedRunningTime="2026-07-27 19:58:49.548401568 +0000 UTC m=+1144.100317985"
Jul 27 19:59:00.532063 ip-10-0-142-140 kubenswrapper[2562]: I0727 19:59:00.532034    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dnrwh"
Jul 27 20:00:00.152520 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:00:00.152469    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753040-p8qqd"]
Jul 27 20:00:00.156107 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:00:00.156092    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-p8qqd"
Jul 27 20:00:00.158163 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:00:00.158145    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x6cbr\""
Jul 27 20:00:00.162311 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:00:00.162278    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753040-p8qqd"]
Jul 27 20:00:00.168249 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:00:00.168217    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-p8qqd"
Jul 27 20:00:00.308826 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:00:00.308795    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753040-p8qqd"]
Jul 27 20:00:00.311534 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:00:00.311506    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6604a889_4fb4_4402_9b7e_3452eb63b2a8.slice/crio-c5c541c185c3025e6e09bb83ed102152d06a439b88d02afdb13af91017103aee WatchSource:0}: Error finding container c5c541c185c3025e6e09bb83ed102152d06a439b88d02afdb13af91017103aee: Status 404 returned error can't find the container with id c5c541c185c3025e6e09bb83ed102152d06a439b88d02afdb13af91017103aee
Jul 27 20:00:00.313403 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:00:00.313386    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:00:00.890286 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:00:00.890241    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-p8qqd" event={"ID":"6604a889-4fb4-4402-9b7e-3452eb63b2a8","Type":"ContainerStarted","Data":"c5c541c185c3025e6e09bb83ed102152d06a439b88d02afdb13af91017103aee"}
Jul 27 20:00:03.908203 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:00:03.908164    2562 generic.go:356] "Generic (PLEG): container finished" podID="6604a889-4fb4-4402-9b7e-3452eb63b2a8" containerID="b9bd1232d9d3d11558372ddda79a0544d361a9e3097e767c6982a06f0046172f" exitCode=0
Jul 27 20:00:03.908594 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:00:03.908250    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-p8qqd" event={"ID":"6604a889-4fb4-4402-9b7e-3452eb63b2a8","Type":"ContainerDied","Data":"b9bd1232d9d3d11558372ddda79a0544d361a9e3097e767c6982a06f0046172f"}
Jul 27 20:00:05.053547 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:00:05.053521    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-p8qqd"
Jul 27 20:00:05.918407 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:00:05.918378    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-p8qqd"
Jul 27 20:00:05.918610 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:00:05.918367    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-p8qqd" event={"ID":"6604a889-4fb4-4402-9b7e-3452eb63b2a8","Type":"ContainerDied","Data":"c5c541c185c3025e6e09bb83ed102152d06a439b88d02afdb13af91017103aee"}
Jul 27 20:00:05.918610 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:00:05.918528    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5c541c185c3025e6e09bb83ed102152d06a439b88d02afdb13af91017103aee"
Jul 27 20:01:42.488981 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.488942    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58b9cbd668-4wfzv"]
Jul 27 20:01:42.489472 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.489345    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6604a889-4fb4-4402-9b7e-3452eb63b2a8" containerName="cleanup"
Jul 27 20:01:42.489472 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.489357    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6604a889-4fb4-4402-9b7e-3452eb63b2a8" containerName="cleanup"
Jul 27 20:01:42.489472 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.489425    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6604a889-4fb4-4402-9b7e-3452eb63b2a8" containerName="cleanup"
Jul 27 20:01:42.493407 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.493387    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.495450 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.495430    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 20:01:42.496050 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.496027    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 20:01:42.496050 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.496044    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 20:01:42.496205 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.496066    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 20:01:42.496205 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.496103    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sj6nc\""
Jul 27 20:01:42.496205 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.496051    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 20:01:42.501911 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.501887    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 20:01:42.507120 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.507101    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b9cbd668-4wfzv"]
Jul 27 20:01:42.571961 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.571926    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c080bc09-35ce-46a3-9b82-280e9c8e3355-console-config\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.572130 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.571966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c080bc09-35ce-46a3-9b82-280e9c8e3355-console-oauth-config\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.572130 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.572054    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c080bc09-35ce-46a3-9b82-280e9c8e3355-trusted-ca-bundle\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.572130 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.572082    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c080bc09-35ce-46a3-9b82-280e9c8e3355-console-serving-cert\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.572130 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.572103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c080bc09-35ce-46a3-9b82-280e9c8e3355-oauth-serving-cert\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.572302 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.572194    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bd6s\" (UniqueName: \"kubernetes.io/projected/c080bc09-35ce-46a3-9b82-280e9c8e3355-kube-api-access-2bd6s\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.572302 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.572222    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c080bc09-35ce-46a3-9b82-280e9c8e3355-service-ca\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.672996 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.672964    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bd6s\" (UniqueName: \"kubernetes.io/projected/c080bc09-35ce-46a3-9b82-280e9c8e3355-kube-api-access-2bd6s\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.672996 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.672999    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c080bc09-35ce-46a3-9b82-280e9c8e3355-service-ca\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.673263 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.673040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c080bc09-35ce-46a3-9b82-280e9c8e3355-console-config\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.673263 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.673065    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c080bc09-35ce-46a3-9b82-280e9c8e3355-console-oauth-config\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.673263 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.673120    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c080bc09-35ce-46a3-9b82-280e9c8e3355-trusted-ca-bundle\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.673263 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.673148    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c080bc09-35ce-46a3-9b82-280e9c8e3355-console-serving-cert\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.673263 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.673174    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c080bc09-35ce-46a3-9b82-280e9c8e3355-oauth-serving-cert\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.673854 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.673825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c080bc09-35ce-46a3-9b82-280e9c8e3355-service-ca\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.673956 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.673849    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c080bc09-35ce-46a3-9b82-280e9c8e3355-console-config\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.673956 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.673884    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c080bc09-35ce-46a3-9b82-280e9c8e3355-oauth-serving-cert\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.674383 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.674361    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c080bc09-35ce-46a3-9b82-280e9c8e3355-trusted-ca-bundle\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.676253 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.676225    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c080bc09-35ce-46a3-9b82-280e9c8e3355-console-oauth-config\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.676746 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.676728    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c080bc09-35ce-46a3-9b82-280e9c8e3355-console-serving-cert\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.681071 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.681047    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bd6s\" (UniqueName: \"kubernetes.io/projected/c080bc09-35ce-46a3-9b82-280e9c8e3355-kube-api-access-2bd6s\") pod \"console-58b9cbd668-4wfzv\" (UID: \"c080bc09-35ce-46a3-9b82-280e9c8e3355\") " pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.803927 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.803830    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:42.944892 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:42.944837    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b9cbd668-4wfzv"]
Jul 27 20:01:42.947237 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:01:42.947209    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc080bc09_35ce_46a3_9b82_280e9c8e3355.slice/crio-bbf82f0c97770be7daeac2123ca12dcba109c110189c39737179512b111194f4 WatchSource:0}: Error finding container bbf82f0c97770be7daeac2123ca12dcba109c110189c39737179512b111194f4: Status 404 returned error can't find the container with id bbf82f0c97770be7daeac2123ca12dcba109c110189c39737179512b111194f4
Jul 27 20:01:43.419339 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:43.419297    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b9cbd668-4wfzv" event={"ID":"c080bc09-35ce-46a3-9b82-280e9c8e3355","Type":"ContainerStarted","Data":"d663625d473f93c2fe067d6cb688b93a7e0077bcd9fe6a6dbcabf0b4053f66fd"}
Jul 27 20:01:43.419561 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:43.419360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b9cbd668-4wfzv" event={"ID":"c080bc09-35ce-46a3-9b82-280e9c8e3355","Type":"ContainerStarted","Data":"bbf82f0c97770be7daeac2123ca12dcba109c110189c39737179512b111194f4"}
Jul 27 20:01:43.448059 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:43.448011    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58b9cbd668-4wfzv" podStartSLOduration=1.447994048 podStartE2EDuration="1.447994048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:01:43.445634094 +0000 UTC m=+1317.997550511" watchObservedRunningTime="2026-07-27 20:01:43.447994048 +0000 UTC m=+1317.999910465"
Jul 27 20:01:52.804628 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:52.804590    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:52.805132 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:52.804654    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:52.809419 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:52.809397    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:01:53.472932 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:01:53.472894    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58b9cbd668-4wfzv"
Jul 27 20:06:03.401082 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.401048    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"]
Jul 27 20:06:03.404851 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.404832    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.407261 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.407238    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7f6e65b7-openshift-default-dockercfg-8ljn8\""
Jul 27 20:06:03.466241 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.466212    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"]
Jul 27 20:06:03.522291 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.522251    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istiod-ca-cert\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.522291 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.522292    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-workload-socket\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.522553 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.522318    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-workload-certs\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.522553 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.522405    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt76v\" (UniqueName: \"kubernetes.io/projected/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-kube-api-access-kt76v\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.522553 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.522435    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-token\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.522553 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.522533    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-envoy\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.522757 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.522621    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-credential-socket\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.522757 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.522656    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-podinfo\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.522757 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.522708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-data\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.624044 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.624010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-credential-socket\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.624044 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.624045    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-podinfo\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.624287 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.624073    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-data\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.624287 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.624100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istiod-ca-cert\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.624287 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.624118    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-workload-socket\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.624287 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.624139    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-workload-certs\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.624287 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.624181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt76v\" (UniqueName: \"kubernetes.io/projected/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-kube-api-access-kt76v\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.624287 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.624203    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-token\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.624287 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.624264    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-envoy\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.624676 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.624523    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-data\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.624676 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.624573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-credential-socket\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.624749 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.624691    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-workload-certs\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.624749 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.624712    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-workload-socket\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.624944 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.624923    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istiod-ca-cert\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.626810 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.626791    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-envoy\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.626936 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.626874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-podinfo\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.633838 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.633815    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-token\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.634054 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.634033    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt76v\" (UniqueName: \"kubernetes.io/projected/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-kube-api-access-kt76v\") pod \"e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") " pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.715903 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.715796    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:03.865352 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.865325    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"]
Jul 27 20:06:03.867123 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:06:03.867091    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf71252f_7efd_4598_ac8f_cc8a4ec6bede.slice/crio-5abd3fbbcb80afd0e17363f17775f2f4749734229009f0cd56af15eb629dbcf6 WatchSource:0}: Error finding container 5abd3fbbcb80afd0e17363f17775f2f4749734229009f0cd56af15eb629dbcf6: Status 404 returned error can't find the container with id 5abd3fbbcb80afd0e17363f17775f2f4749734229009f0cd56af15eb629dbcf6
Jul 27 20:06:03.869766 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.869732    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:06:03.869855 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.869800    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:06:03.869855 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:03.869828    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:06:04.802048 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:04.802008    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td" event={"ID":"cf71252f-7efd-4598-ac8f-cc8a4ec6bede","Type":"ContainerStarted","Data":"40ea7fad193ec145afe068f232ed8ede27a3120bb16b271bf896f36556869287"}
Jul 27 20:06:04.802442 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:04.802064    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td" event={"ID":"cf71252f-7efd-4598-ac8f-cc8a4ec6bede","Type":"ContainerStarted","Data":"5abd3fbbcb80afd0e17363f17775f2f4749734229009f0cd56af15eb629dbcf6"}
Jul 27 20:06:04.824120 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:04.824065    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td" podStartSLOduration=1.824044663 podStartE2EDuration="1.824044663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:04.822790848 +0000 UTC m=+1579.374707266" watchObservedRunningTime="2026-07-27 20:06:04.824044663 +0000 UTC m=+1579.375961082"
Jul 27 20:06:05.716593 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:05.716556    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:05.721499 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:05.721451    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:05.808183 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:05.808147    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:05.809117 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:05.809099    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:06.262503 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.262454    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"]
Jul 27 20:06:06.266854 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.266836    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:06.270879 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.270854    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7f6e65b7-serving-cert\""
Jul 27 20:06:06.283381 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.283358    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"]
Jul 27 20:06:06.350082 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.350050    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vg7n\" (UniqueName: \"kubernetes.io/projected/655dfd3e-5874-4853-8a1d-935b2471b860-kube-api-access-4vg7n\") pod \"maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4\" (UID: \"655dfd3e-5874-4853-8a1d-935b2471b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:06.350269 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.350104    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655dfd3e-5874-4853-8a1d-935b2471b860-maas-api-tls\") pod \"maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4\" (UID: \"655dfd3e-5874-4853-8a1d-935b2471b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:06.350269 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.350203    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655dfd3e-5874-4853-8a1d-935b2471b860-openshift-service-ca\") pod \"maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4\" (UID: \"655dfd3e-5874-4853-8a1d-935b2471b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:06.451120 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.451083    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655dfd3e-5874-4853-8a1d-935b2471b860-openshift-service-ca\") pod \"maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4\" (UID: \"655dfd3e-5874-4853-8a1d-935b2471b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:06.451333 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.451204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vg7n\" (UniqueName: \"kubernetes.io/projected/655dfd3e-5874-4853-8a1d-935b2471b860-kube-api-access-4vg7n\") pod \"maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4\" (UID: \"655dfd3e-5874-4853-8a1d-935b2471b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:06.451333 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.451265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655dfd3e-5874-4853-8a1d-935b2471b860-maas-api-tls\") pod \"maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4\" (UID: \"655dfd3e-5874-4853-8a1d-935b2471b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:06.451859 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.451838    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655dfd3e-5874-4853-8a1d-935b2471b860-openshift-service-ca\") pod \"maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4\" (UID: \"655dfd3e-5874-4853-8a1d-935b2471b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:06.453781 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.453762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655dfd3e-5874-4853-8a1d-935b2471b860-maas-api-tls\") pod \"maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4\" (UID: \"655dfd3e-5874-4853-8a1d-935b2471b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:06.462168 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.462144    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vg7n\" (UniqueName: \"kubernetes.io/projected/655dfd3e-5874-4853-8a1d-935b2471b860-kube-api-access-4vg7n\") pod \"maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4\" (UID: \"655dfd3e-5874-4853-8a1d-935b2471b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:06.579270 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.579182    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:06.724996 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.724972    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"]
Jul 27 20:06:06.727354 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:06:06.727309    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655dfd3e_5874_4853_8a1d_935b2471b860.slice/crio-eccacba9c06464a9105c0d459ca2d1856e418e540428a20e2a8a69ebbabde1bc WatchSource:0}: Error finding container eccacba9c06464a9105c0d459ca2d1856e418e540428a20e2a8a69ebbabde1bc: Status 404 returned error can't find the container with id eccacba9c06464a9105c0d459ca2d1856e418e540428a20e2a8a69ebbabde1bc
Jul 27 20:06:06.728901 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.728882    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:06:06.814506 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:06.814443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4" event={"ID":"655dfd3e-5874-4853-8a1d-935b2471b860","Type":"ContainerStarted","Data":"eccacba9c06464a9105c0d459ca2d1856e418e540428a20e2a8a69ebbabde1bc"}
Jul 27 20:06:07.821813 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:07.821780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4" event={"ID":"655dfd3e-5874-4853-8a1d-935b2471b860","Type":"ContainerStarted","Data":"fcab433105d6ba3fecfb64a3b50cd14a947e705afc35ef4319930f4358c50d65"}
Jul 27 20:06:07.822244 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:07.821865    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:07.841283 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:07.841228    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4" podStartSLOduration=0.955082493 podStartE2EDuration="1.841209687s" totalImagesPullingTime="886.127194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:07.839054692 +0000 UTC m=+1582.390971112" watchObservedRunningTime="2026-07-27 20:06:07.841209687 +0000 UTC m=+1582.393126108"
Jul 27 20:06:13.833804 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:13.833727    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:18.883651 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:18.883616    2562 generic.go:356] "Generic (PLEG): container finished" podID="cf71252f-7efd-4598-ac8f-cc8a4ec6bede" containerID="40ea7fad193ec145afe068f232ed8ede27a3120bb16b271bf896f36556869287" exitCode=0
Jul 27 20:06:18.884348 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:18.883692    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td" event={"ID":"cf71252f-7efd-4598-ac8f-cc8a4ec6bede","Type":"ContainerDied","Data":"40ea7fad193ec145afe068f232ed8ede27a3120bb16b271bf896f36556869287"}
Jul 27 20:06:18.884348 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:18.884152    2562 scope.go:117] "RemoveContainer" containerID="40ea7fad193ec145afe068f232ed8ede27a3120bb16b271bf896f36556869287"
Jul 27 20:06:18.885308 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:18.885278    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:06:18.885405 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:18.885349    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:06:18.885405 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:18.885383    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:06:19.909568 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:19.909518    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td" event={"ID":"cf71252f-7efd-4598-ac8f-cc8a4ec6bede","Type":"ContainerStarted","Data":"f2da25f40c0b0125d2c763bac00732bacb6c16788aa27f656a1ded5a6f1fa824"}
Jul 27 20:06:20.716139 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:20.716106    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:20.717208 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:20.717185    2562 patch_prober.go:28] interesting pod/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 27 20:06:20.717293 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:20.717247    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td" podUID="cf71252f-7efd-4598-ac8f-cc8a4ec6bede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 27 20:06:20.808251 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:20.808214    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:21.716756 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:21.716713    2562 patch_prober.go:28] interesting pod/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 27 20:06:21.717233 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:21.716789    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td" podUID="cf71252f-7efd-4598-ac8f-cc8a4ec6bede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 27 20:06:22.716275 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:22.716238    2562 patch_prober.go:28] interesting pod/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 27 20:06:22.716467 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:22.716305    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td" podUID="cf71252f-7efd-4598-ac8f-cc8a4ec6bede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 27 20:06:23.716652 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:23.716606    2562 patch_prober.go:28] interesting pod/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 27 20:06:23.717146 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:23.716709    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td" podUID="cf71252f-7efd-4598-ac8f-cc8a4ec6bede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 27 20:06:24.720756 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:24.720725    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:24.721879 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:24.721857    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:42.376354 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:42.376268    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"]
Jul 27 20:06:42.376941 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:42.376566    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4" podUID="655dfd3e-5874-4853-8a1d-935b2471b860" containerName="maas-api" containerID="cri-o://fcab433105d6ba3fecfb64a3b50cd14a947e705afc35ef4319930f4358c50d65" gracePeriod=30
Jul 27 20:06:42.645044 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:42.644967    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:42.710716 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:42.710688    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655dfd3e-5874-4853-8a1d-935b2471b860-maas-api-tls\") pod \"655dfd3e-5874-4853-8a1d-935b2471b860\" (UID: \"655dfd3e-5874-4853-8a1d-935b2471b860\") "
Jul 27 20:06:42.710716 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:42.710755    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vg7n\" (UniqueName: \"kubernetes.io/projected/655dfd3e-5874-4853-8a1d-935b2471b860-kube-api-access-4vg7n\") pod \"655dfd3e-5874-4853-8a1d-935b2471b860\" (UID: \"655dfd3e-5874-4853-8a1d-935b2471b860\") "
Jul 27 20:06:42.711022 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:42.710811    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655dfd3e-5874-4853-8a1d-935b2471b860-openshift-service-ca\") pod \"655dfd3e-5874-4853-8a1d-935b2471b860\" (UID: \"655dfd3e-5874-4853-8a1d-935b2471b860\") "
Jul 27 20:06:42.711255 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:42.711225    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655dfd3e-5874-4853-8a1d-935b2471b860-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "655dfd3e-5874-4853-8a1d-935b2471b860" (UID: "655dfd3e-5874-4853-8a1d-935b2471b860"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:06:42.713197 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:42.713176    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655dfd3e-5874-4853-8a1d-935b2471b860-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "655dfd3e-5874-4853-8a1d-935b2471b860" (UID: "655dfd3e-5874-4853-8a1d-935b2471b860"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:06:42.713303 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:42.713242    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655dfd3e-5874-4853-8a1d-935b2471b860-kube-api-access-4vg7n" (OuterVolumeSpecName: "kube-api-access-4vg7n") pod "655dfd3e-5874-4853-8a1d-935b2471b860" (UID: "655dfd3e-5874-4853-8a1d-935b2471b860"). InnerVolumeSpecName "kube-api-access-4vg7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:06:42.812130 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:42.812094    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655dfd3e-5874-4853-8a1d-935b2471b860-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:06:42.812130 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:42.812124    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vg7n\" (UniqueName: \"kubernetes.io/projected/655dfd3e-5874-4853-8a1d-935b2471b860-kube-api-access-4vg7n\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:06:42.812130 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:42.812133    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655dfd3e-5874-4853-8a1d-935b2471b860-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:06:43.030388 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:43.030291    2562 generic.go:356] "Generic (PLEG): container finished" podID="655dfd3e-5874-4853-8a1d-935b2471b860" containerID="fcab433105d6ba3fecfb64a3b50cd14a947e705afc35ef4319930f4358c50d65" exitCode=0
Jul 27 20:06:43.030388 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:43.030337    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"
Jul 27 20:06:43.030388 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:43.030342    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4" event={"ID":"655dfd3e-5874-4853-8a1d-935b2471b860","Type":"ContainerDied","Data":"fcab433105d6ba3fecfb64a3b50cd14a947e705afc35ef4319930f4358c50d65"}
Jul 27 20:06:43.030682 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:43.030396    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4" event={"ID":"655dfd3e-5874-4853-8a1d-935b2471b860","Type":"ContainerDied","Data":"eccacba9c06464a9105c0d459ca2d1856e418e540428a20e2a8a69ebbabde1bc"}
Jul 27 20:06:43.030682 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:43.030423    2562 scope.go:117] "RemoveContainer" containerID="fcab433105d6ba3fecfb64a3b50cd14a947e705afc35ef4319930f4358c50d65"
Jul 27 20:06:43.039695 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:43.039678    2562 scope.go:117] "RemoveContainer" containerID="fcab433105d6ba3fecfb64a3b50cd14a947e705afc35ef4319930f4358c50d65"
Jul 27 20:06:43.039991 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:06:43.039972    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcab433105d6ba3fecfb64a3b50cd14a947e705afc35ef4319930f4358c50d65\": container with ID starting with fcab433105d6ba3fecfb64a3b50cd14a947e705afc35ef4319930f4358c50d65 not found: ID does not exist" containerID="fcab433105d6ba3fecfb64a3b50cd14a947e705afc35ef4319930f4358c50d65"
Jul 27 20:06:43.040051 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:43.040002    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcab433105d6ba3fecfb64a3b50cd14a947e705afc35ef4319930f4358c50d65"} err="failed to get container status \"fcab433105d6ba3fecfb64a3b50cd14a947e705afc35ef4319930f4358c50d65\": rpc error: code = NotFound desc = could not find container \"fcab433105d6ba3fecfb64a3b50cd14a947e705afc35ef4319930f4358c50d65\": container with ID starting with fcab433105d6ba3fecfb64a3b50cd14a947e705afc35ef4319930f4358c50d65 not found: ID does not exist"
Jul 27 20:06:43.051290 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:43.051267    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"]
Jul 27 20:06:43.056142 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:43.056118    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7f6e65b7-6bdfdf446f-kdgw4"]
Jul 27 20:06:43.992249 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:43.992213    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="655dfd3e-5874-4853-8a1d-935b2471b860" path="/var/lib/kubelet/pods/655dfd3e-5874-4853-8a1d-935b2471b860/volumes"
Jul 27 20:06:46.933816 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:46.933783    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"]
Jul 27 20:06:46.934207 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:46.934002    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td" podUID="cf71252f-7efd-4598-ac8f-cc8a4ec6bede" containerName="istio-proxy" containerID="cri-o://f2da25f40c0b0125d2c763bac00732bacb6c16788aa27f656a1ded5a6f1fa824" gracePeriod=30
Jul 27 20:06:47.913813 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.913780    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"]
Jul 27 20:06:47.914380 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.914364    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655dfd3e-5874-4853-8a1d-935b2471b860" containerName="maas-api"
Jul 27 20:06:47.914380 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.914382    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="655dfd3e-5874-4853-8a1d-935b2471b860" containerName="maas-api"
Jul 27 20:06:47.914502 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.914454    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="655dfd3e-5874-4853-8a1d-935b2471b860" containerName="maas-api"
Jul 27 20:06:47.918883 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.918861    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:47.921610 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.921587    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f658717b-gw-openshift-default-dockercfg-sl766\""
Jul 27 20:06:47.932199 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.932176    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"]
Jul 27 20:06:47.963775 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.963734    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szrlp\" (UniqueName: \"kubernetes.io/projected/f75107bc-a737-4b42-853a-5545287068c6-kube-api-access-szrlp\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:47.963775 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.963775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-istio-data\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:47.964296 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.963869    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f75107bc-a737-4b42-853a-5545287068c6-istio-podinfo\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:47.964296 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.963905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f75107bc-a737-4b42-853a-5545287068c6-istio-token\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:47.964296 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.963932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-istio-envoy\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:47.964296 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.963970    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-workload-socket\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:47.964296 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.963998    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f75107bc-a737-4b42-853a-5545287068c6-istiod-ca-cert\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:47.964296 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.964068    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-credential-socket\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:47.964296 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:47.964104    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-workload-certs\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.065360 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.065330    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrlp\" (UniqueName: \"kubernetes.io/projected/f75107bc-a737-4b42-853a-5545287068c6-kube-api-access-szrlp\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.065609 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.065371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-istio-data\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.065609 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.065416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f75107bc-a737-4b42-853a-5545287068c6-istio-podinfo\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.065609 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.065436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f75107bc-a737-4b42-853a-5545287068c6-istio-token\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.065609 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.065452    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-istio-envoy\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.065609 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.065514    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-workload-socket\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.065609 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.065551    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f75107bc-a737-4b42-853a-5545287068c6-istiod-ca-cert\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.065912 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.065624    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-credential-socket\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.065912 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.065725    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-workload-certs\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.066023 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.065952    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-istio-data\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.066023 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.065982    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-workload-socket\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.066102 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.066051    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-workload-certs\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.066102 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.066068    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-credential-socket\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.066332 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.066312    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f75107bc-a737-4b42-853a-5545287068c6-istiod-ca-cert\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.068029 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.068012    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-istio-envoy\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.068115 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.068070    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f75107bc-a737-4b42-853a-5545287068c6-istio-podinfo\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.080305 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.080285    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szrlp\" (UniqueName: \"kubernetes.io/projected/f75107bc-a737-4b42-853a-5545287068c6-kube-api-access-szrlp\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.080698 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.080682    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f75107bc-a737-4b42-853a-5545287068c6-istio-token\") pod \"e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") " pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.229318 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.229229    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:48.384961 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.384913    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"]
Jul 27 20:06:48.390508 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.390408    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:06:48.390631 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.390581    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:06:48.390851 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:48.390818    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:06:49.065707 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:49.065671    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx" event={"ID":"f75107bc-a737-4b42-853a-5545287068c6","Type":"ContainerStarted","Data":"04fad153819298d2aeb5d0ec16961f124e70842115c2a8e0d8693c4c602edf00"}
Jul 27 20:06:49.066192 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:49.065737    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx" event={"ID":"f75107bc-a737-4b42-853a-5545287068c6","Type":"ContainerStarted","Data":"e1b5c7e4106e0d6d85b4c6c35478cd3ce23b3c53fd2f0264d918f833c170019c"}
Jul 27 20:06:49.100055 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:49.100000    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx" podStartSLOduration=2.09998473 podStartE2EDuration="2.09998473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:49.097682997 +0000 UTC m=+1623.649599429" watchObservedRunningTime="2026-07-27 20:06:49.09998473 +0000 UTC m=+1623.651901148"
Jul 27 20:06:49.230453 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:49.230419    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:49.235339 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:49.235314    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:50.071091 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:50.071060    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:50.072255 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:50.072237    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:06:50.670229 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:50.670193    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"]
Jul 27 20:06:50.674770 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:50.674744    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:06:50.677350 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:50.677317    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f658717b-serving-cert\""
Jul 27 20:06:50.686922 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:50.686861    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"]
Jul 27 20:06:50.793300 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:50.793263    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbc6a337-220d-4879-a25a-3bbc4f09799d-maas-api-tls\") pod \"maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k\" (UID: \"dbc6a337-220d-4879-a25a-3bbc4f09799d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:06:50.793526 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:50.793339    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7n57\" (UniqueName: \"kubernetes.io/projected/dbc6a337-220d-4879-a25a-3bbc4f09799d-kube-api-access-j7n57\") pod \"maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k\" (UID: \"dbc6a337-220d-4879-a25a-3bbc4f09799d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:06:50.793526 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:50.793388    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbc6a337-220d-4879-a25a-3bbc4f09799d-openshift-service-ca\") pod \"maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k\" (UID: \"dbc6a337-220d-4879-a25a-3bbc4f09799d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:06:50.894994 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:50.894951    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7n57\" (UniqueName: \"kubernetes.io/projected/dbc6a337-220d-4879-a25a-3bbc4f09799d-kube-api-access-j7n57\") pod \"maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k\" (UID: \"dbc6a337-220d-4879-a25a-3bbc4f09799d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:06:50.895174 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:50.895020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbc6a337-220d-4879-a25a-3bbc4f09799d-openshift-service-ca\") pod \"maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k\" (UID: \"dbc6a337-220d-4879-a25a-3bbc4f09799d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:06:50.895174 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:50.895116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbc6a337-220d-4879-a25a-3bbc4f09799d-maas-api-tls\") pod \"maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k\" (UID: \"dbc6a337-220d-4879-a25a-3bbc4f09799d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:06:50.895257 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:06:50.895240    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-serving-cert: secret "maas-api-e2e-migrate-f658717b-serving-cert" not found
Jul 27 20:06:50.895327 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:06:50.895316    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbc6a337-220d-4879-a25a-3bbc4f09799d-maas-api-tls podName:dbc6a337-220d-4879-a25a-3bbc4f09799d nodeName:}" failed. No retries permitted until 2026-07-27 20:06:51.395295478 +0000 UTC m=+1625.947211877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dbc6a337-220d-4879-a25a-3bbc4f09799d-maas-api-tls") pod "maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k" (UID: "dbc6a337-220d-4879-a25a-3bbc4f09799d") : secret "maas-api-e2e-migrate-f658717b-serving-cert" not found
Jul 27 20:06:50.895652 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:50.895633    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbc6a337-220d-4879-a25a-3bbc4f09799d-openshift-service-ca\") pod \"maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k\" (UID: \"dbc6a337-220d-4879-a25a-3bbc4f09799d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:06:50.905046 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:50.905022    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7n57\" (UniqueName: \"kubernetes.io/projected/dbc6a337-220d-4879-a25a-3bbc4f09799d-kube-api-access-j7n57\") pod \"maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k\" (UID: \"dbc6a337-220d-4879-a25a-3bbc4f09799d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:06:51.399988 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:51.399956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbc6a337-220d-4879-a25a-3bbc4f09799d-maas-api-tls\") pod \"maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k\" (UID: \"dbc6a337-220d-4879-a25a-3bbc4f09799d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:06:51.402613 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:51.402584    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbc6a337-220d-4879-a25a-3bbc4f09799d-maas-api-tls\") pod \"maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k\" (UID: \"dbc6a337-220d-4879-a25a-3bbc4f09799d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:06:51.592023 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:51.591986    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:06:51.764574 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:51.764539    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"]
Jul 27 20:06:51.765890 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:06:51.765867    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc6a337_220d_4879_a25a_3bbc4f09799d.slice/crio-51bee6ca00c9d3f9bcde94ce4ddaf0e2605485b58cf220fa193a248a39b5e612 WatchSource:0}: Error finding container 51bee6ca00c9d3f9bcde94ce4ddaf0e2605485b58cf220fa193a248a39b5e612: Status 404 returned error can't find the container with id 51bee6ca00c9d3f9bcde94ce4ddaf0e2605485b58cf220fa193a248a39b5e612
Jul 27 20:06:52.083147 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.083110    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k" event={"ID":"dbc6a337-220d-4879-a25a-3bbc4f09799d","Type":"ContainerStarted","Data":"51bee6ca00c9d3f9bcde94ce4ddaf0e2605485b58cf220fa193a248a39b5e612"}
Jul 27 20:06:52.085603 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.085565    2562 generic.go:356] "Generic (PLEG): container finished" podID="cf71252f-7efd-4598-ac8f-cc8a4ec6bede" containerID="f2da25f40c0b0125d2c763bac00732bacb6c16788aa27f656a1ded5a6f1fa824" exitCode=0
Jul 27 20:06:52.085976 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.085790    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td" event={"ID":"cf71252f-7efd-4598-ac8f-cc8a4ec6bede","Type":"ContainerDied","Data":"f2da25f40c0b0125d2c763bac00732bacb6c16788aa27f656a1ded5a6f1fa824"}
Jul 27 20:06:52.085976 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.085866    2562 scope.go:117] "RemoveContainer" containerID="40ea7fad193ec145afe068f232ed8ede27a3120bb16b271bf896f36556869287"
Jul 27 20:06:52.196852 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.196783    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:52.316141 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316109    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-token\") pod \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") "
Jul 27 20:06:52.316271 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316159    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-envoy\") pod \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") "
Jul 27 20:06:52.316271 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316205    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-data\") pod \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") "
Jul 27 20:06:52.316271 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316234    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-workload-socket\") pod \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") "
Jul 27 20:06:52.316271 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316253    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-workload-certs\") pod \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") "
Jul 27 20:06:52.316512 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316295    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-credential-socket\") pod \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") "
Jul 27 20:06:52.316512 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316340    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-podinfo\") pod \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") "
Jul 27 20:06:52.316512 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316386    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt76v\" (UniqueName: \"kubernetes.io/projected/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-kube-api-access-kt76v\") pod \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") "
Jul 27 20:06:52.316512 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316455    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istiod-ca-cert\") pod \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\" (UID: \"cf71252f-7efd-4598-ac8f-cc8a4ec6bede\") "
Jul 27 20:06:52.316727 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316522    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf71252f-7efd-4598-ac8f-cc8a4ec6bede" (UID: "cf71252f-7efd-4598-ac8f-cc8a4ec6bede"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:06:52.316727 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316563    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf71252f-7efd-4598-ac8f-cc8a4ec6bede" (UID: "cf71252f-7efd-4598-ac8f-cc8a4ec6bede"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:06:52.316727 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316609    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf71252f-7efd-4598-ac8f-cc8a4ec6bede" (UID: "cf71252f-7efd-4598-ac8f-cc8a4ec6bede"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:06:52.316954 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316895    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf71252f-7efd-4598-ac8f-cc8a4ec6bede" (UID: "cf71252f-7efd-4598-ac8f-cc8a4ec6bede"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:06:52.316954 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316907    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-workload-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:06:52.317068 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.316985    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-workload-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:06:52.317068 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.317057    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-credential-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:06:52.317348 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.317212    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf71252f-7efd-4598-ac8f-cc8a4ec6bede" (UID: "cf71252f-7efd-4598-ac8f-cc8a4ec6bede"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:06:52.318684 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.318659    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf71252f-7efd-4598-ac8f-cc8a4ec6bede" (UID: "cf71252f-7efd-4598-ac8f-cc8a4ec6bede"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:06:52.318763 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.318717    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-kube-api-access-kt76v" (OuterVolumeSpecName: "kube-api-access-kt76v") pod "cf71252f-7efd-4598-ac8f-cc8a4ec6bede" (UID: "cf71252f-7efd-4598-ac8f-cc8a4ec6bede"). InnerVolumeSpecName "kube-api-access-kt76v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:06:52.318969 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.318951    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf71252f-7efd-4598-ac8f-cc8a4ec6bede" (UID: "cf71252f-7efd-4598-ac8f-cc8a4ec6bede"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:06:52.319020 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.319007    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf71252f-7efd-4598-ac8f-cc8a4ec6bede" (UID: "cf71252f-7efd-4598-ac8f-cc8a4ec6bede"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:06:52.418352 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.418313    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istiod-ca-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:06:52.418352 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.418342    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:06:52.418352 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.418351    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-envoy\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:06:52.418352 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.418358    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-data\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:06:52.418352 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.418365    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-istio-podinfo\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:06:52.418880 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:52.418373    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt76v\" (UniqueName: \"kubernetes.io/projected/cf71252f-7efd-4598-ac8f-cc8a4ec6bede-kube-api-access-kt76v\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:06:53.092864 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:53.092820    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k" event={"ID":"dbc6a337-220d-4879-a25a-3bbc4f09799d","Type":"ContainerStarted","Data":"07c1de1eb9cb0ada5afd74d1f0ca003080170a6d74adf6d74182ebe0916dc263"}
Jul 27 20:06:53.093070 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:53.093016    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:06:53.094135 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:53.094107    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"
Jul 27 20:06:53.094273 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:53.094131    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td" event={"ID":"cf71252f-7efd-4598-ac8f-cc8a4ec6bede","Type":"ContainerDied","Data":"5abd3fbbcb80afd0e17363f17775f2f4749734229009f0cd56af15eb629dbcf6"}
Jul 27 20:06:53.094273 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:53.094184    2562 scope.go:117] "RemoveContainer" containerID="f2da25f40c0b0125d2c763bac00732bacb6c16788aa27f656a1ded5a6f1fa824"
Jul 27 20:06:53.113131 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:53.113093    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k" podStartSLOduration=2.666284667 podStartE2EDuration="3.113079281s" totalImagesPullingTime="446.794614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:53.110700665 +0000 UTC m=+1627.662617084" watchObservedRunningTime="2026-07-27 20:06:53.113079281 +0000 UTC m=+1627.664995699"
Jul 27 20:06:53.125799 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:53.125774    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"]
Jul 27 20:06:53.131913 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:53.131892    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7f6e65b7-openshift-default-695dbb868-tl7td"]
Jul 27 20:06:53.991888 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:53.991855    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf71252f-7efd-4598-ac8f-cc8a4ec6bede" path="/var/lib/kubelet/pods/cf71252f-7efd-4598-ac8f-cc8a4ec6bede/volumes"
Jul 27 20:06:59.107088 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:06:59.107061    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:07:05.072355 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:05.072275    2562 patch_prober.go:28] interesting pod/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 27 20:07:05.072355 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:05.072333    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx" podUID="f75107bc-a737-4b42-853a-5545287068c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 27 20:07:05.167399 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:05.167312    2562 generic.go:356] "Generic (PLEG): container finished" podID="f75107bc-a737-4b42-853a-5545287068c6" containerID="04fad153819298d2aeb5d0ec16961f124e70842115c2a8e0d8693c4c602edf00" exitCode=0
Jul 27 20:07:05.167587 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:05.167387    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx" event={"ID":"f75107bc-a737-4b42-853a-5545287068c6","Type":"ContainerDied","Data":"04fad153819298d2aeb5d0ec16961f124e70842115c2a8e0d8693c4c602edf00"}
Jul 27 20:07:05.167864 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:05.167848    2562 scope.go:117] "RemoveContainer" containerID="04fad153819298d2aeb5d0ec16961f124e70842115c2a8e0d8693c4c602edf00"
Jul 27 20:07:05.168966 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:05.168935    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:07:05.169046 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:05.169006    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:07:05.169046 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:05.169037    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:07:06.175688 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:06.175646    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx" event={"ID":"f75107bc-a737-4b42-853a-5545287068c6","Type":"ContainerStarted","Data":"d4ca6bf950e926e268551485fd1e276e46093b8dd53a30acf9d57a8cc105c7ed"}
Jul 27 20:07:06.229627 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:06.229589    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:07:06.229794 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:06.229762    2562 patch_prober.go:28] interesting pod/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 27 20:07:06.229839 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:06.229807    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx" podUID="f75107bc-a737-4b42-853a-5545287068c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 27 20:07:07.230492 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:07.230427    2562 patch_prober.go:28] interesting pod/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 27 20:07:07.230973 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:07.230533    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx" podUID="f75107bc-a737-4b42-853a-5545287068c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 27 20:07:08.230143 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:08.230109    2562 patch_prober.go:28] interesting pod/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 27 20:07:08.230316 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:08.230171    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx" podUID="f75107bc-a737-4b42-853a-5545287068c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 27 20:07:09.229967 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:09.229927    2562 patch_prober.go:28] interesting pod/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 27 20:07:09.230424 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:09.229986    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx" podUID="f75107bc-a737-4b42-853a-5545287068c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 27 20:07:10.230432 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:10.230244    2562 patch_prober.go:28] interesting pod/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 27 20:07:10.230921 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:10.230533    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx" podUID="f75107bc-a737-4b42-853a-5545287068c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 27 20:07:11.233625 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:11.233595    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:07:11.233997 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:11.233863    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:07:11.234416 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:11.234402    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:07:26.115866 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:26.115833    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"]
Jul 27 20:07:26.116278 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:26.116050    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k" podUID="dbc6a337-220d-4879-a25a-3bbc4f09799d" containerName="maas-api" containerID="cri-o://07c1de1eb9cb0ada5afd74d1f0ca003080170a6d74adf6d74182ebe0916dc263" gracePeriod=30
Jul 27 20:07:26.305829 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:26.305799    2562 generic.go:356] "Generic (PLEG): container finished" podID="dbc6a337-220d-4879-a25a-3bbc4f09799d" containerID="07c1de1eb9cb0ada5afd74d1f0ca003080170a6d74adf6d74182ebe0916dc263" exitCode=0
Jul 27 20:07:26.305968 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:26.305845    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k" event={"ID":"dbc6a337-220d-4879-a25a-3bbc4f09799d","Type":"ContainerDied","Data":"07c1de1eb9cb0ada5afd74d1f0ca003080170a6d74adf6d74182ebe0916dc263"}
Jul 27 20:07:26.385214 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:26.385188    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:07:26.449936 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:26.449901    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbc6a337-220d-4879-a25a-3bbc4f09799d-openshift-service-ca\") pod \"dbc6a337-220d-4879-a25a-3bbc4f09799d\" (UID: \"dbc6a337-220d-4879-a25a-3bbc4f09799d\") "
Jul 27 20:07:26.450122 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:26.449963    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7n57\" (UniqueName: \"kubernetes.io/projected/dbc6a337-220d-4879-a25a-3bbc4f09799d-kube-api-access-j7n57\") pod \"dbc6a337-220d-4879-a25a-3bbc4f09799d\" (UID: \"dbc6a337-220d-4879-a25a-3bbc4f09799d\") "
Jul 27 20:07:26.450122 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:26.450007    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbc6a337-220d-4879-a25a-3bbc4f09799d-maas-api-tls\") pod \"dbc6a337-220d-4879-a25a-3bbc4f09799d\" (UID: \"dbc6a337-220d-4879-a25a-3bbc4f09799d\") "
Jul 27 20:07:26.450267 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:26.450242    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc6a337-220d-4879-a25a-3bbc4f09799d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dbc6a337-220d-4879-a25a-3bbc4f09799d" (UID: "dbc6a337-220d-4879-a25a-3bbc4f09799d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:26.450362 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:26.450350    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbc6a337-220d-4879-a25a-3bbc4f09799d-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:07:26.452331 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:26.452309    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc6a337-220d-4879-a25a-3bbc4f09799d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dbc6a337-220d-4879-a25a-3bbc4f09799d" (UID: "dbc6a337-220d-4879-a25a-3bbc4f09799d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:26.452331 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:26.452311    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc6a337-220d-4879-a25a-3bbc4f09799d-kube-api-access-j7n57" (OuterVolumeSpecName: "kube-api-access-j7n57") pod "dbc6a337-220d-4879-a25a-3bbc4f09799d" (UID: "dbc6a337-220d-4879-a25a-3bbc4f09799d"). InnerVolumeSpecName "kube-api-access-j7n57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:07:26.551014 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:26.550982    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7n57\" (UniqueName: \"kubernetes.io/projected/dbc6a337-220d-4879-a25a-3bbc4f09799d-kube-api-access-j7n57\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:07:26.551014 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:26.551009    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbc6a337-220d-4879-a25a-3bbc4f09799d-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:07:27.312457 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:27.312431    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"
Jul 27 20:07:27.312457 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:27.312441    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k" event={"ID":"dbc6a337-220d-4879-a25a-3bbc4f09799d","Type":"ContainerDied","Data":"51bee6ca00c9d3f9bcde94ce4ddaf0e2605485b58cf220fa193a248a39b5e612"}
Jul 27 20:07:27.312967 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:27.312519    2562 scope.go:117] "RemoveContainer" containerID="07c1de1eb9cb0ada5afd74d1f0ca003080170a6d74adf6d74182ebe0916dc263"
Jul 27 20:07:27.334862 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:27.334806    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"]
Jul 27 20:07:27.337099 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:27.337077    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f658717b-86757bb69d-sqh7k"]
Jul 27 20:07:27.991894 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:27.991849    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbc6a337-220d-4879-a25a-3bbc4f09799d" path="/var/lib/kubelet/pods/dbc6a337-220d-4879-a25a-3bbc4f09799d/volumes"
Jul 27 20:07:30.752661 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:30.752621    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"]
Jul 27 20:07:30.753229 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:30.752888    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx" podUID="f75107bc-a737-4b42-853a-5545287068c6" containerName="istio-proxy" containerID="cri-o://d4ca6bf950e926e268551485fd1e276e46093b8dd53a30acf9d57a8cc105c7ed" gracePeriod=30
Jul 27 20:07:31.511928 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.511894    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"]
Jul 27 20:07:31.512427 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.512410    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf71252f-7efd-4598-ac8f-cc8a4ec6bede" containerName="istio-proxy"
Jul 27 20:07:31.512584 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.512429    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf71252f-7efd-4598-ac8f-cc8a4ec6bede" containerName="istio-proxy"
Jul 27 20:07:31.512584 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.512440    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc6a337-220d-4879-a25a-3bbc4f09799d" containerName="maas-api"
Jul 27 20:07:31.512584 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.512449    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc6a337-220d-4879-a25a-3bbc4f09799d" containerName="maas-api"
Jul 27 20:07:31.512584 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.512546    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf71252f-7efd-4598-ac8f-cc8a4ec6bede" containerName="istio-proxy"
Jul 27 20:07:31.512584 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.512555    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf71252f-7efd-4598-ac8f-cc8a4ec6bede" containerName="istio-proxy"
Jul 27 20:07:31.512871 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.512698    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf71252f-7efd-4598-ac8f-cc8a4ec6bede" containerName="istio-proxy"
Jul 27 20:07:31.512871 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.512713    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf71252f-7efd-4598-ac8f-cc8a4ec6bede" containerName="istio-proxy"
Jul 27 20:07:31.512871 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.512723    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc6a337-220d-4879-a25a-3bbc4f09799d" containerName="maas-api"
Jul 27 20:07:31.517327 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.517303    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.522051 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.522031    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e3c0fa8a-openshift-default-dockercfg-jm8tr\""
Jul 27 20:07:31.529684 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.529661    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"]
Jul 27 20:07:31.601706 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.601676    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-istio-envoy\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.601905 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.601740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b931e1d-bd27-480b-8444-f999433d79e9-istio-token\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.601905 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.601803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b931e1d-bd27-480b-8444-f999433d79e9-istio-podinfo\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.601905 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.601844    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-workload-certs\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.602085 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.601931    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-credential-socket\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.602085 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.601975    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-istio-data\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.602085 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.602022    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn295\" (UniqueName: \"kubernetes.io/projected/3b931e1d-bd27-480b-8444-f999433d79e9-kube-api-access-dn295\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.602085 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.602056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-workload-socket\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.602085 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.602081    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b931e1d-bd27-480b-8444-f999433d79e9-istiod-ca-cert\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.702801 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.702767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b931e1d-bd27-480b-8444-f999433d79e9-istio-podinfo\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.702801 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.702819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-workload-certs\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.703066 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.702858    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-credential-socket\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.703066 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.702877    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-istio-data\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.703066 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.702900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn295\" (UniqueName: \"kubernetes.io/projected/3b931e1d-bd27-480b-8444-f999433d79e9-kube-api-access-dn295\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.703066 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.702927    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-workload-socket\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.703066 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.702944    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b931e1d-bd27-480b-8444-f999433d79e9-istiod-ca-cert\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.703066 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.702966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-istio-envoy\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.703066 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.703033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b931e1d-bd27-480b-8444-f999433d79e9-istio-token\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.703457 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.703302    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-credential-socket\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.703457 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.703354    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-workload-certs\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.703595 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.703529    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-workload-socket\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.703650 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.703593    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-istio-data\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.703874 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.703853    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b931e1d-bd27-480b-8444-f999433d79e9-istiod-ca-cert\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.705668 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.705648    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-istio-envoy\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.705811 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.705785    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b931e1d-bd27-480b-8444-f999433d79e9-istio-podinfo\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.711113 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.711085    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b931e1d-bd27-480b-8444-f999433d79e9-istio-token\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.711372 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.711357    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn295\" (UniqueName: \"kubernetes.io/projected/3b931e1d-bd27-480b-8444-f999433d79e9-kube-api-access-dn295\") pod \"e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") " pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.830928 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.830827    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:31.980899 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.980860    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"]
Jul 27 20:07:31.982465 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:07:31.982434    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b931e1d_bd27_480b_8444_f999433d79e9.slice/crio-a28974e062313b0d8dfa83ccc3dace4621e0def28adf902bc1296b521ab59911 WatchSource:0}: Error finding container a28974e062313b0d8dfa83ccc3dace4621e0def28adf902bc1296b521ab59911: Status 404 returned error can't find the container with id a28974e062313b0d8dfa83ccc3dace4621e0def28adf902bc1296b521ab59911
Jul 27 20:07:31.984935 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.984900    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:07:31.985049 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.985012    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:07:31.985109 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:31.985070    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:07:32.341372 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:32.341334    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9" event={"ID":"3b931e1d-bd27-480b-8444-f999433d79e9","Type":"ContainerStarted","Data":"0194ebff5882a6cacc97fe1f255634e632364267c9e464a1b225c9752b46cbbb"}
Jul 27 20:07:32.341595 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:32.341385    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9" event={"ID":"3b931e1d-bd27-480b-8444-f999433d79e9","Type":"ContainerStarted","Data":"a28974e062313b0d8dfa83ccc3dace4621e0def28adf902bc1296b521ab59911"}
Jul 27 20:07:32.363874 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:32.363825    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9" podStartSLOduration=1.3638084240000001 podStartE2EDuration="1.363808424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:07:32.361319712 +0000 UTC m=+1666.913236129" watchObservedRunningTime="2026-07-27 20:07:32.363808424 +0000 UTC m=+1666.915724842"
Jul 27 20:07:32.831616 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:32.831577    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:33.836833 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:33.836799    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:33.848576 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:33.848529    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"]
Jul 27 20:07:33.852285 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:33.852267    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:07:33.854366 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:33.854342    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e3c0fa8a-serving-cert\""
Jul 27 20:07:33.869055 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:33.869022    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"]
Jul 27 20:07:33.927282 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:33.927228    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc38978-807d-42c3-8246-5398bd24d992-openshift-service-ca\") pod \"maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46\" (UID: \"adc38978-807d-42c3-8246-5398bd24d992\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:07:33.927499 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:33.927392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7n2w\" (UniqueName: \"kubernetes.io/projected/adc38978-807d-42c3-8246-5398bd24d992-kube-api-access-h7n2w\") pod \"maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46\" (UID: \"adc38978-807d-42c3-8246-5398bd24d992\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:07:33.927579 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:33.927514    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adc38978-807d-42c3-8246-5398bd24d992-maas-api-tls\") pod \"maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46\" (UID: \"adc38978-807d-42c3-8246-5398bd24d992\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:07:34.028045 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:34.028008    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7n2w\" (UniqueName: \"kubernetes.io/projected/adc38978-807d-42c3-8246-5398bd24d992-kube-api-access-h7n2w\") pod \"maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46\" (UID: \"adc38978-807d-42c3-8246-5398bd24d992\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:07:34.028224 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:34.028089    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adc38978-807d-42c3-8246-5398bd24d992-maas-api-tls\") pod \"maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46\" (UID: \"adc38978-807d-42c3-8246-5398bd24d992\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:07:34.028224 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:34.028128    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc38978-807d-42c3-8246-5398bd24d992-openshift-service-ca\") pod \"maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46\" (UID: \"adc38978-807d-42c3-8246-5398bd24d992\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:07:34.028301 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:07:34.028243    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-serving-cert: secret "maas-api-e2e-ait-e3c0fa8a-serving-cert" not found
Jul 27 20:07:34.028336 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:07:34.028314    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adc38978-807d-42c3-8246-5398bd24d992-maas-api-tls podName:adc38978-807d-42c3-8246-5398bd24d992 nodeName:}" failed. No retries permitted until 2026-07-27 20:07:34.528297948 +0000 UTC m=+1669.080214344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/adc38978-807d-42c3-8246-5398bd24d992-maas-api-tls") pod "maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46" (UID: "adc38978-807d-42c3-8246-5398bd24d992") : secret "maas-api-e2e-ait-e3c0fa8a-serving-cert" not found
Jul 27 20:07:34.028853 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:34.028835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc38978-807d-42c3-8246-5398bd24d992-openshift-service-ca\") pod \"maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46\" (UID: \"adc38978-807d-42c3-8246-5398bd24d992\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:07:34.037327 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:34.037297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7n2w\" (UniqueName: \"kubernetes.io/projected/adc38978-807d-42c3-8246-5398bd24d992-kube-api-access-h7n2w\") pod \"maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46\" (UID: \"adc38978-807d-42c3-8246-5398bd24d992\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:07:34.352518 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:34.352465    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:34.353472 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:34.353452    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:34.533135 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:34.533104    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adc38978-807d-42c3-8246-5398bd24d992-maas-api-tls\") pod \"maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46\" (UID: \"adc38978-807d-42c3-8246-5398bd24d992\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:07:34.535835 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:34.535814    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adc38978-807d-42c3-8246-5398bd24d992-maas-api-tls\") pod \"maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46\" (UID: \"adc38978-807d-42c3-8246-5398bd24d992\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:07:34.763549 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:34.763495    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:07:35.164676 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:35.164644    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"]
Jul 27 20:07:35.168198 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:07:35.168166    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc38978_807d_42c3_8246_5398bd24d992.slice/crio-bd4ed906edbdecbc124a682515e1bd6e316fc4f2af0b02c2fbfa04c10b63ff61 WatchSource:0}: Error finding container bd4ed906edbdecbc124a682515e1bd6e316fc4f2af0b02c2fbfa04c10b63ff61: Status 404 returned error can't find the container with id bd4ed906edbdecbc124a682515e1bd6e316fc4f2af0b02c2fbfa04c10b63ff61
Jul 27 20:07:35.359377 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:35.359343    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46" event={"ID":"adc38978-807d-42c3-8246-5398bd24d992","Type":"ContainerStarted","Data":"bd4ed906edbdecbc124a682515e1bd6e316fc4f2af0b02c2fbfa04c10b63ff61"}
Jul 27 20:07:36.017765 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.017738    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:07:36.150508 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.150399    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-credential-socket\") pod \"f75107bc-a737-4b42-853a-5545287068c6\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") "
Jul 27 20:07:36.150508 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.150465    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szrlp\" (UniqueName: \"kubernetes.io/projected/f75107bc-a737-4b42-853a-5545287068c6-kube-api-access-szrlp\") pod \"f75107bc-a737-4b42-853a-5545287068c6\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") "
Jul 27 20:07:36.150717 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.150520    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f75107bc-a737-4b42-853a-5545287068c6-istio-token\") pod \"f75107bc-a737-4b42-853a-5545287068c6\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") "
Jul 27 20:07:36.150717 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.150591    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-istio-data\") pod \"f75107bc-a737-4b42-853a-5545287068c6\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") "
Jul 27 20:07:36.150717 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.150611    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f75107bc-a737-4b42-853a-5545287068c6-istio-podinfo\") pod \"f75107bc-a737-4b42-853a-5545287068c6\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") "
Jul 27 20:07:36.150717 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.150629    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-workload-certs\") pod \"f75107bc-a737-4b42-853a-5545287068c6\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") "
Jul 27 20:07:36.150717 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.150648    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-workload-socket\") pod \"f75107bc-a737-4b42-853a-5545287068c6\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") "
Jul 27 20:07:36.150717 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.150691    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f75107bc-a737-4b42-853a-5545287068c6-istiod-ca-cert\") pod \"f75107bc-a737-4b42-853a-5545287068c6\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") "
Jul 27 20:07:36.151024 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.150739    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f75107bc-a737-4b42-853a-5545287068c6" (UID: "f75107bc-a737-4b42-853a-5545287068c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:36.151024 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.150776    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-istio-envoy\") pod \"f75107bc-a737-4b42-853a-5545287068c6\" (UID: \"f75107bc-a737-4b42-853a-5545287068c6\") "
Jul 27 20:07:36.151024 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.151010    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f75107bc-a737-4b42-853a-5545287068c6" (UID: "f75107bc-a737-4b42-853a-5545287068c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:36.151174 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.151094    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f75107bc-a737-4b42-853a-5545287068c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f75107bc-a737-4b42-853a-5545287068c6" (UID: "f75107bc-a737-4b42-853a-5545287068c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:36.151174 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.151141    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-workload-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:07:36.151174 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.151161    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-credential-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:07:36.151332 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.151245    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f75107bc-a737-4b42-853a-5545287068c6" (UID: "f75107bc-a737-4b42-853a-5545287068c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:36.151386 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.151368    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "f75107bc-a737-4b42-853a-5545287068c6" (UID: "f75107bc-a737-4b42-853a-5545287068c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:36.153250 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.153224    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f75107bc-a737-4b42-853a-5545287068c6" (UID: "f75107bc-a737-4b42-853a-5545287068c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:36.153250 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.153233    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f75107bc-a737-4b42-853a-5545287068c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "f75107bc-a737-4b42-853a-5545287068c6" (UID: "f75107bc-a737-4b42-853a-5545287068c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:07:36.153560 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.153539    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f75107bc-a737-4b42-853a-5545287068c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f75107bc-a737-4b42-853a-5545287068c6" (UID: "f75107bc-a737-4b42-853a-5545287068c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:07:36.153560 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.153548    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f75107bc-a737-4b42-853a-5545287068c6-kube-api-access-szrlp" (OuterVolumeSpecName: "kube-api-access-szrlp") pod "f75107bc-a737-4b42-853a-5545287068c6" (UID: "f75107bc-a737-4b42-853a-5545287068c6"). InnerVolumeSpecName "kube-api-access-szrlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:07:36.252076 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.252040    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-istio-data\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:07:36.252076 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.252068    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f75107bc-a737-4b42-853a-5545287068c6-istio-podinfo\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:07:36.252076 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.252077    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-workload-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:07:36.252076 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.252086    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f75107bc-a737-4b42-853a-5545287068c6-istiod-ca-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:07:36.252593 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.252094    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f75107bc-a737-4b42-853a-5545287068c6-istio-envoy\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:07:36.252593 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.252104    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szrlp\" (UniqueName: \"kubernetes.io/projected/f75107bc-a737-4b42-853a-5545287068c6-kube-api-access-szrlp\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:07:36.252593 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.252114    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f75107bc-a737-4b42-853a-5545287068c6-istio-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:07:36.366997 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.366959    2562 generic.go:356] "Generic (PLEG): container finished" podID="f75107bc-a737-4b42-853a-5545287068c6" containerID="d4ca6bf950e926e268551485fd1e276e46093b8dd53a30acf9d57a8cc105c7ed" exitCode=0
Jul 27 20:07:36.367173 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.367006    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"
Jul 27 20:07:36.367173 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.367042    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx" event={"ID":"f75107bc-a737-4b42-853a-5545287068c6","Type":"ContainerDied","Data":"d4ca6bf950e926e268551485fd1e276e46093b8dd53a30acf9d57a8cc105c7ed"}
Jul 27 20:07:36.367173 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.367092    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx" event={"ID":"f75107bc-a737-4b42-853a-5545287068c6","Type":"ContainerDied","Data":"e1b5c7e4106e0d6d85b4c6c35478cd3ce23b3c53fd2f0264d918f833c170019c"}
Jul 27 20:07:36.367173 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.367116    2562 scope.go:117] "RemoveContainer" containerID="d4ca6bf950e926e268551485fd1e276e46093b8dd53a30acf9d57a8cc105c7ed"
Jul 27 20:07:36.369395 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.369234    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46" event={"ID":"adc38978-807d-42c3-8246-5398bd24d992","Type":"ContainerStarted","Data":"02de55af7b74ae9eae76157b97af02fc51429918c45ec809219f8e64ba6e9457"}
Jul 27 20:07:36.369395 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.369319    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:07:36.378142 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.378123    2562 scope.go:117] "RemoveContainer" containerID="04fad153819298d2aeb5d0ec16961f124e70842115c2a8e0d8693c4c602edf00"
Jul 27 20:07:36.389435 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.389342    2562 scope.go:117] "RemoveContainer" containerID="d4ca6bf950e926e268551485fd1e276e46093b8dd53a30acf9d57a8cc105c7ed"
Jul 27 20:07:36.392238 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.392193    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46" podStartSLOduration=3.003363637 podStartE2EDuration="3.392178121s" totalImagesPullingTime="388.814484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:07:36.389578858 +0000 UTC m=+1670.941495278" watchObservedRunningTime="2026-07-27 20:07:36.392178121 +0000 UTC m=+1670.944094541"
Jul 27 20:07:36.393287 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:07:36.393246    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ca6bf950e926e268551485fd1e276e46093b8dd53a30acf9d57a8cc105c7ed\": container with ID starting with d4ca6bf950e926e268551485fd1e276e46093b8dd53a30acf9d57a8cc105c7ed not found: ID does not exist" containerID="d4ca6bf950e926e268551485fd1e276e46093b8dd53a30acf9d57a8cc105c7ed"
Jul 27 20:07:36.393380 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.393296    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ca6bf950e926e268551485fd1e276e46093b8dd53a30acf9d57a8cc105c7ed"} err="failed to get container status \"d4ca6bf950e926e268551485fd1e276e46093b8dd53a30acf9d57a8cc105c7ed\": rpc error: code = NotFound desc = could not find container \"d4ca6bf950e926e268551485fd1e276e46093b8dd53a30acf9d57a8cc105c7ed\": container with ID starting with d4ca6bf950e926e268551485fd1e276e46093b8dd53a30acf9d57a8cc105c7ed not found: ID does not exist"
Jul 27 20:07:36.393380 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.393318    2562 scope.go:117] "RemoveContainer" containerID="04fad153819298d2aeb5d0ec16961f124e70842115c2a8e0d8693c4c602edf00"
Jul 27 20:07:36.393597 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:07:36.393578    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fad153819298d2aeb5d0ec16961f124e70842115c2a8e0d8693c4c602edf00\": container with ID starting with 04fad153819298d2aeb5d0ec16961f124e70842115c2a8e0d8693c4c602edf00 not found: ID does not exist" containerID="04fad153819298d2aeb5d0ec16961f124e70842115c2a8e0d8693c4c602edf00"
Jul 27 20:07:36.393646 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.393622    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fad153819298d2aeb5d0ec16961f124e70842115c2a8e0d8693c4c602edf00"} err="failed to get container status \"04fad153819298d2aeb5d0ec16961f124e70842115c2a8e0d8693c4c602edf00\": rpc error: code = NotFound desc = could not find container \"04fad153819298d2aeb5d0ec16961f124e70842115c2a8e0d8693c4c602edf00\": container with ID starting with 04fad153819298d2aeb5d0ec16961f124e70842115c2a8e0d8693c4c602edf00 not found: ID does not exist"
Jul 27 20:07:36.407576 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.407494    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"]
Jul 27 20:07:36.413385 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:36.413363    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f658717b-gw-openshift-default-658f78fcf6-sw7fx"]
Jul 27 20:07:37.993855 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:37.993818    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f75107bc-a737-4b42-853a-5545287068c6" path="/var/lib/kubelet/pods/f75107bc-a737-4b42-853a-5545287068c6/volumes"
Jul 27 20:07:42.388255 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:42.388215    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:07:48.451216 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:48.451120    2562 generic.go:356] "Generic (PLEG): container finished" podID="3b931e1d-bd27-480b-8444-f999433d79e9" containerID="0194ebff5882a6cacc97fe1f255634e632364267c9e464a1b225c9752b46cbbb" exitCode=0
Jul 27 20:07:48.451216 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:48.451194    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9" event={"ID":"3b931e1d-bd27-480b-8444-f999433d79e9","Type":"ContainerDied","Data":"0194ebff5882a6cacc97fe1f255634e632364267c9e464a1b225c9752b46cbbb"}
Jul 27 20:07:48.451724 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:48.451662    2562 scope.go:117] "RemoveContainer" containerID="0194ebff5882a6cacc97fe1f255634e632364267c9e464a1b225c9752b46cbbb"
Jul 27 20:07:48.452884 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:48.452850    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:07:48.452966 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:48.452939    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:07:48.453006 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:48.452976    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:07:49.353273 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:49.353239    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:49.458861 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:49.458826    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9" event={"ID":"3b931e1d-bd27-480b-8444-f999433d79e9","Type":"ContainerStarted","Data":"185437234b6a7a9d7a8a00f762371979d982d423f317e1b3bb1bdf9aaf5f98b8"}
Jul 27 20:07:49.831556 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:49.831523    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:49.832821 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:49.832793    2562 patch_prober.go:28] interesting pod/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 27 20:07:49.832930 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:49.832846    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9" podUID="3b931e1d-bd27-480b-8444-f999433d79e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 27 20:07:50.831558 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:50.831522    2562 patch_prober.go:28] interesting pod/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 27 20:07:50.831943 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:50.831578    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9" podUID="3b931e1d-bd27-480b-8444-f999433d79e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 27 20:07:51.832078 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:51.832036    2562 patch_prober.go:28] interesting pod/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 27 20:07:51.832518 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:51.832124    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9" podUID="3b931e1d-bd27-480b-8444-f999433d79e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 27 20:07:52.832064 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:52.832027    2562 patch_prober.go:28] interesting pod/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 27 20:07:52.832243 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:52.832081    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9" podUID="3b931e1d-bd27-480b-8444-f999433d79e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 27 20:07:53.836332 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:53.836301    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:53.836750 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:53.836607    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:07:53.837296 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:07:53.837282    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:08:09.998048 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:09.997962    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"]
Jul 27 20:08:09.998820 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:09.998207    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46" podUID="adc38978-807d-42c3-8246-5398bd24d992" containerName="maas-api" containerID="cri-o://02de55af7b74ae9eae76157b97af02fc51429918c45ec809219f8e64ba6e9457" gracePeriod=30
Jul 27 20:08:10.259461 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.259397    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:08:10.392371 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.392325    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7n2w\" (UniqueName: \"kubernetes.io/projected/adc38978-807d-42c3-8246-5398bd24d992-kube-api-access-h7n2w\") pod \"adc38978-807d-42c3-8246-5398bd24d992\" (UID: \"adc38978-807d-42c3-8246-5398bd24d992\") "
Jul 27 20:08:10.392371 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.392373    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc38978-807d-42c3-8246-5398bd24d992-openshift-service-ca\") pod \"adc38978-807d-42c3-8246-5398bd24d992\" (UID: \"adc38978-807d-42c3-8246-5398bd24d992\") "
Jul 27 20:08:10.392653 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.392458    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adc38978-807d-42c3-8246-5398bd24d992-maas-api-tls\") pod \"adc38978-807d-42c3-8246-5398bd24d992\" (UID: \"adc38978-807d-42c3-8246-5398bd24d992\") "
Jul 27 20:08:10.392802 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.392777    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc38978-807d-42c3-8246-5398bd24d992-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "adc38978-807d-42c3-8246-5398bd24d992" (UID: "adc38978-807d-42c3-8246-5398bd24d992"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:08:10.394758 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.394727    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc38978-807d-42c3-8246-5398bd24d992-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "adc38978-807d-42c3-8246-5398bd24d992" (UID: "adc38978-807d-42c3-8246-5398bd24d992"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:08:10.394850 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.394754    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc38978-807d-42c3-8246-5398bd24d992-kube-api-access-h7n2w" (OuterVolumeSpecName: "kube-api-access-h7n2w") pod "adc38978-807d-42c3-8246-5398bd24d992" (UID: "adc38978-807d-42c3-8246-5398bd24d992"). InnerVolumeSpecName "kube-api-access-h7n2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:08:10.494169 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.494137    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adc38978-807d-42c3-8246-5398bd24d992-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:10.494169 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.494164    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7n2w\" (UniqueName: \"kubernetes.io/projected/adc38978-807d-42c3-8246-5398bd24d992-kube-api-access-h7n2w\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:10.494169 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.494174    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc38978-807d-42c3-8246-5398bd24d992-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:10.587760 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.587670    2562 generic.go:356] "Generic (PLEG): container finished" podID="adc38978-807d-42c3-8246-5398bd24d992" containerID="02de55af7b74ae9eae76157b97af02fc51429918c45ec809219f8e64ba6e9457" exitCode=0
Jul 27 20:08:10.587760 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.587728    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"
Jul 27 20:08:10.587941 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.587750    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46" event={"ID":"adc38978-807d-42c3-8246-5398bd24d992","Type":"ContainerDied","Data":"02de55af7b74ae9eae76157b97af02fc51429918c45ec809219f8e64ba6e9457"}
Jul 27 20:08:10.587941 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.587819    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46" event={"ID":"adc38978-807d-42c3-8246-5398bd24d992","Type":"ContainerDied","Data":"bd4ed906edbdecbc124a682515e1bd6e316fc4f2af0b02c2fbfa04c10b63ff61"}
Jul 27 20:08:10.587941 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.587849    2562 scope.go:117] "RemoveContainer" containerID="02de55af7b74ae9eae76157b97af02fc51429918c45ec809219f8e64ba6e9457"
Jul 27 20:08:10.597255 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.597233    2562 scope.go:117] "RemoveContainer" containerID="02de55af7b74ae9eae76157b97af02fc51429918c45ec809219f8e64ba6e9457"
Jul 27 20:08:10.597578 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:08:10.597558    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02de55af7b74ae9eae76157b97af02fc51429918c45ec809219f8e64ba6e9457\": container with ID starting with 02de55af7b74ae9eae76157b97af02fc51429918c45ec809219f8e64ba6e9457 not found: ID does not exist" containerID="02de55af7b74ae9eae76157b97af02fc51429918c45ec809219f8e64ba6e9457"
Jul 27 20:08:10.597681 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.597586    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02de55af7b74ae9eae76157b97af02fc51429918c45ec809219f8e64ba6e9457"} err="failed to get container status \"02de55af7b74ae9eae76157b97af02fc51429918c45ec809219f8e64ba6e9457\": rpc error: code = NotFound desc = could not find container \"02de55af7b74ae9eae76157b97af02fc51429918c45ec809219f8e64ba6e9457\": container with ID starting with 02de55af7b74ae9eae76157b97af02fc51429918c45ec809219f8e64ba6e9457 not found: ID does not exist"
Jul 27 20:08:10.608185 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.608160    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"]
Jul 27 20:08:10.614598 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:10.614577    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3c0fa8a-7f897694cd-bjt46"]
Jul 27 20:08:11.991931 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:11.991894    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adc38978-807d-42c3-8246-5398bd24d992" path="/var/lib/kubelet/pods/adc38978-807d-42c3-8246-5398bd24d992/volumes"
Jul 27 20:08:15.797543 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:15.797513    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"]
Jul 27 20:08:15.798067 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:15.797727    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9" podUID="3b931e1d-bd27-480b-8444-f999433d79e9" containerName="istio-proxy" containerID="cri-o://185437234b6a7a9d7a8a00f762371979d982d423f317e1b3bb1bdf9aaf5f98b8" gracePeriod=30
Jul 27 20:08:16.570517 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.570461    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"]
Jul 27 20:08:16.571097 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.571072    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f75107bc-a737-4b42-853a-5545287068c6" containerName="istio-proxy"
Jul 27 20:08:16.571241 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.571216    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f75107bc-a737-4b42-853a-5545287068c6" containerName="istio-proxy"
Jul 27 20:08:16.571314 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.571284    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adc38978-807d-42c3-8246-5398bd24d992" containerName="maas-api"
Jul 27 20:08:16.571314 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.571295    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="adc38978-807d-42c3-8246-5398bd24d992" containerName="maas-api"
Jul 27 20:08:16.571432 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.571417    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="adc38978-807d-42c3-8246-5398bd24d992" containerName="maas-api"
Jul 27 20:08:16.571522 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.571440    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f75107bc-a737-4b42-853a-5545287068c6" containerName="istio-proxy"
Jul 27 20:08:16.571522 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.571454    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f75107bc-a737-4b42-853a-5545287068c6" containerName="istio-proxy"
Jul 27 20:08:16.574692 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.574671    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.577157 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.577095    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-269943c6-openshift-default-dockercfg-xs5dj\""
Jul 27 20:08:16.587001 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.586978    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"]
Jul 27 20:08:16.654330 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.654290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-workload-socket\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.654330 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.654338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dd910b-b601-4769-8a22-33f9583390a5-istiod-ca-cert\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.654630 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.654365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-workload-certs\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.654630 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.654431    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-istio-envoy\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.654630 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.654462    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59dd910b-b601-4769-8a22-33f9583390a5-istio-token\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.654630 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.654555    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-istio-data\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.654630 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.654584    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dd910b-b601-4769-8a22-33f9583390a5-istio-podinfo\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.654823 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.654650    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6w7\" (UniqueName: \"kubernetes.io/projected/59dd910b-b601-4769-8a22-33f9583390a5-kube-api-access-8c6w7\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.654823 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.654688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-credential-socket\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.755982 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.755939    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-workload-certs\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.756182 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.755996    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-istio-envoy\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.756182 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.756018    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59dd910b-b601-4769-8a22-33f9583390a5-istio-token\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.756182 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.756065    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-istio-data\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.756182 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.756103    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dd910b-b601-4769-8a22-33f9583390a5-istio-podinfo\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.756182 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.756145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6w7\" (UniqueName: \"kubernetes.io/projected/59dd910b-b601-4769-8a22-33f9583390a5-kube-api-access-8c6w7\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.756462 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.756200    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-credential-socket\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.756462 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.756232    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-workload-socket\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.756462 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.756277    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dd910b-b601-4769-8a22-33f9583390a5-istiod-ca-cert\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.756462 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.756409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-workload-certs\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.756725 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.756511    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-istio-data\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.756725 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.756658    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-credential-socket\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.756839 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.756738    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-workload-socket\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.757031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.757009    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dd910b-b601-4769-8a22-33f9583390a5-istiod-ca-cert\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.759166 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.759132    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-istio-envoy\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.759432 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.759410    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dd910b-b601-4769-8a22-33f9583390a5-istio-podinfo\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.765704 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.765680    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59dd910b-b601-4769-8a22-33f9583390a5-istio-token\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.766031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.766012    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6w7\" (UniqueName: \"kubernetes.io/projected/59dd910b-b601-4769-8a22-33f9583390a5-kube-api-access-8c6w7\") pod \"e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") " pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:16.887607 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:16.887570    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:17.042855 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:17.042827    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"]
Jul 27 20:08:17.044037 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:08:17.044002    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59dd910b_b601_4769_8a22_33f9583390a5.slice/crio-2320e693497488ff04453dfd853328e8ee23fe59b9b0547666e56ad296659194 WatchSource:0}: Error finding container 2320e693497488ff04453dfd853328e8ee23fe59b9b0547666e56ad296659194: Status 404 returned error can't find the container with id 2320e693497488ff04453dfd853328e8ee23fe59b9b0547666e56ad296659194
Jul 27 20:08:17.046291 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:17.046257    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:08:17.046362 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:17.046325    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:08:17.046362 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:17.046352    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:08:17.628064 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:17.628024    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f" event={"ID":"59dd910b-b601-4769-8a22-33f9583390a5","Type":"ContainerStarted","Data":"ee3097240eb21613550bf5845bfa1992f3b29531b9768edf59b19d03f8e460ce"}
Jul 27 20:08:17.628245 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:17.628126    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f" event={"ID":"59dd910b-b601-4769-8a22-33f9583390a5","Type":"ContainerStarted","Data":"2320e693497488ff04453dfd853328e8ee23fe59b9b0547666e56ad296659194"}
Jul 27 20:08:17.649250 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:17.649185    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f" podStartSLOduration=1.6491645080000001 podStartE2EDuration="1.649164508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:08:17.647615599 +0000 UTC m=+1712.199532017" watchObservedRunningTime="2026-07-27 20:08:17.649164508 +0000 UTC m=+1712.201080928"
Jul 27 20:08:17.888662 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:17.888577    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:18.893777 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:18.893745    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:19.154821 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.154741    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"]
Jul 27 20:08:19.155184 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.155173    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f75107bc-a737-4b42-853a-5545287068c6" containerName="istio-proxy"
Jul 27 20:08:19.155230 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.155186    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f75107bc-a737-4b42-853a-5545287068c6" containerName="istio-proxy"
Jul 27 20:08:19.158543 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.158523    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:19.160791 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.160772    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-269943c6-serving-cert\""
Jul 27 20:08:19.169460 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.169436    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"]
Jul 27 20:08:19.280318 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.280281    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2aeb3e9-1661-47da-9b58-38973a5beaa7-openshift-service-ca\") pod \"maas-api-e2e-derive-269943c6-85df6c4878-5pjtr\" (UID: \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:19.280573 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.280352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8css8\" (UniqueName: \"kubernetes.io/projected/c2aeb3e9-1661-47da-9b58-38973a5beaa7-kube-api-access-8css8\") pod \"maas-api-e2e-derive-269943c6-85df6c4878-5pjtr\" (UID: \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:19.280573 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.280506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2aeb3e9-1661-47da-9b58-38973a5beaa7-maas-api-tls\") pod \"maas-api-e2e-derive-269943c6-85df6c4878-5pjtr\" (UID: \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:19.381124 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.381086    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8css8\" (UniqueName: \"kubernetes.io/projected/c2aeb3e9-1661-47da-9b58-38973a5beaa7-kube-api-access-8css8\") pod \"maas-api-e2e-derive-269943c6-85df6c4878-5pjtr\" (UID: \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:19.381350 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.381156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2aeb3e9-1661-47da-9b58-38973a5beaa7-maas-api-tls\") pod \"maas-api-e2e-derive-269943c6-85df6c4878-5pjtr\" (UID: \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:19.381350 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.381262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2aeb3e9-1661-47da-9b58-38973a5beaa7-openshift-service-ca\") pod \"maas-api-e2e-derive-269943c6-85df6c4878-5pjtr\" (UID: \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:19.381959 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.381939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2aeb3e9-1661-47da-9b58-38973a5beaa7-openshift-service-ca\") pod \"maas-api-e2e-derive-269943c6-85df6c4878-5pjtr\" (UID: \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:19.383838 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.383816    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2aeb3e9-1661-47da-9b58-38973a5beaa7-maas-api-tls\") pod \"maas-api-e2e-derive-269943c6-85df6c4878-5pjtr\" (UID: \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:19.389955 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.389927    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8css8\" (UniqueName: \"kubernetes.io/projected/c2aeb3e9-1661-47da-9b58-38973a5beaa7-kube-api-access-8css8\") pod \"maas-api-e2e-derive-269943c6-85df6c4878-5pjtr\" (UID: \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:19.470907 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.470827    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:19.638789 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.638754    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:19.639910 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.639885    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:19.823982 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:19.823957    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"]
Jul 27 20:08:19.826239 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:08:19.826210    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2aeb3e9_1661_47da_9b58_38973a5beaa7.slice/crio-c3af37e52e96167b6e7f0a8b1b9b5c4f6cdd9342d3e0dc044dab766bfbeb9ab7 WatchSource:0}: Error finding container c3af37e52e96167b6e7f0a8b1b9b5c4f6cdd9342d3e0dc044dab766bfbeb9ab7: Status 404 returned error can't find the container with id c3af37e52e96167b6e7f0a8b1b9b5c4f6cdd9342d3e0dc044dab766bfbeb9ab7
Jul 27 20:08:20.647048 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:20.647003    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr" event={"ID":"c2aeb3e9-1661-47da-9b58-38973a5beaa7","Type":"ContainerStarted","Data":"9f0e33c6ffa5174661b58ad48dd3380af58da0d162aac16d6cacc3785309998c"}
Jul 27 20:08:20.647465 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:20.647065    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr" event={"ID":"c2aeb3e9-1661-47da-9b58-38973a5beaa7","Type":"ContainerStarted","Data":"c3af37e52e96167b6e7f0a8b1b9b5c4f6cdd9342d3e0dc044dab766bfbeb9ab7"}
Jul 27 20:08:20.647465 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:20.647097    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:20.667784 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:20.667734    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr" podStartSLOduration=1.224342325 podStartE2EDuration="1.667719855s" totalImagesPullingTime="443.37753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:08:20.665687229 +0000 UTC m=+1715.217603649" watchObservedRunningTime="2026-07-27 20:08:20.667719855 +0000 UTC m=+1715.219636274"
Jul 27 20:08:21.063008 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.062983    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:08:21.200461 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.200361    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b931e1d-bd27-480b-8444-f999433d79e9-istio-token\") pod \"3b931e1d-bd27-480b-8444-f999433d79e9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") "
Jul 27 20:08:21.200461 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.200408    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b931e1d-bd27-480b-8444-f999433d79e9-istiod-ca-cert\") pod \"3b931e1d-bd27-480b-8444-f999433d79e9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") "
Jul 27 20:08:21.200461 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.200444    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-istio-data\") pod \"3b931e1d-bd27-480b-8444-f999433d79e9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") "
Jul 27 20:08:21.200776 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.200528    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-workload-certs\") pod \"3b931e1d-bd27-480b-8444-f999433d79e9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") "
Jul 27 20:08:21.200776 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.200563    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b931e1d-bd27-480b-8444-f999433d79e9-istio-podinfo\") pod \"3b931e1d-bd27-480b-8444-f999433d79e9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") "
Jul 27 20:08:21.200776 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.200615    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-istio-envoy\") pod \"3b931e1d-bd27-480b-8444-f999433d79e9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") "
Jul 27 20:08:21.200776 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.200646    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-credential-socket\") pod \"3b931e1d-bd27-480b-8444-f999433d79e9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") "
Jul 27 20:08:21.200776 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.200718    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn295\" (UniqueName: \"kubernetes.io/projected/3b931e1d-bd27-480b-8444-f999433d79e9-kube-api-access-dn295\") pod \"3b931e1d-bd27-480b-8444-f999433d79e9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") "
Jul 27 20:08:21.200776 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.200751    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-workload-socket\") pod \"3b931e1d-bd27-480b-8444-f999433d79e9\" (UID: \"3b931e1d-bd27-480b-8444-f999433d79e9\") "
Jul 27 20:08:21.201073 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.200826    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b931e1d-bd27-480b-8444-f999433d79e9" (UID: "3b931e1d-bd27-480b-8444-f999433d79e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:08:21.201073 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.200914    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b931e1d-bd27-480b-8444-f999433d79e9" (UID: "3b931e1d-bd27-480b-8444-f999433d79e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:08:21.201073 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.200859    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b931e1d-bd27-480b-8444-f999433d79e9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b931e1d-bd27-480b-8444-f999433d79e9" (UID: "3b931e1d-bd27-480b-8444-f999433d79e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:08:21.201225 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.201167    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b931e1d-bd27-480b-8444-f999433d79e9" (UID: "3b931e1d-bd27-480b-8444-f999433d79e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:08:21.201225 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.201177    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b931e1d-bd27-480b-8444-f999433d79e9-istiod-ca-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:21.201225 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.201200    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-workload-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:21.201225 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.201213    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-credential-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:21.201714 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.201687    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b931e1d-bd27-480b-8444-f999433d79e9" (UID: "3b931e1d-bd27-480b-8444-f999433d79e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:08:21.202862 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.202841    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b931e1d-bd27-480b-8444-f999433d79e9-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b931e1d-bd27-480b-8444-f999433d79e9" (UID: "3b931e1d-bd27-480b-8444-f999433d79e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:08:21.203013 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.202989    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b931e1d-bd27-480b-8444-f999433d79e9" (UID: "3b931e1d-bd27-480b-8444-f999433d79e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:08:21.203411 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.203381    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b931e1d-bd27-480b-8444-f999433d79e9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b931e1d-bd27-480b-8444-f999433d79e9" (UID: "3b931e1d-bd27-480b-8444-f999433d79e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:08:21.203537 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.203443    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b931e1d-bd27-480b-8444-f999433d79e9-kube-api-access-dn295" (OuterVolumeSpecName: "kube-api-access-dn295") pod "3b931e1d-bd27-480b-8444-f999433d79e9" (UID: "3b931e1d-bd27-480b-8444-f999433d79e9"). InnerVolumeSpecName "kube-api-access-dn295". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:08:21.301981 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.301949    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b931e1d-bd27-480b-8444-f999433d79e9-istio-podinfo\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:21.301981 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.301976    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-istio-envoy\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:21.301981 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.301986    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn295\" (UniqueName: \"kubernetes.io/projected/3b931e1d-bd27-480b-8444-f999433d79e9-kube-api-access-dn295\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:21.302219 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.301997    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-workload-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:21.302219 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.302006    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b931e1d-bd27-480b-8444-f999433d79e9-istio-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:21.302219 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.302014    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b931e1d-bd27-480b-8444-f999433d79e9-istio-data\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:21.654126 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.654092    2562 generic.go:356] "Generic (PLEG): container finished" podID="3b931e1d-bd27-480b-8444-f999433d79e9" containerID="185437234b6a7a9d7a8a00f762371979d982d423f317e1b3bb1bdf9aaf5f98b8" exitCode=0
Jul 27 20:08:21.654623 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.654161    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"
Jul 27 20:08:21.654623 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.654181    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9" event={"ID":"3b931e1d-bd27-480b-8444-f999433d79e9","Type":"ContainerDied","Data":"185437234b6a7a9d7a8a00f762371979d982d423f317e1b3bb1bdf9aaf5f98b8"}
Jul 27 20:08:21.654623 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.654267    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9" event={"ID":"3b931e1d-bd27-480b-8444-f999433d79e9","Type":"ContainerDied","Data":"a28974e062313b0d8dfa83ccc3dace4621e0def28adf902bc1296b521ab59911"}
Jul 27 20:08:21.654623 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.654290    2562 scope.go:117] "RemoveContainer" containerID="185437234b6a7a9d7a8a00f762371979d982d423f317e1b3bb1bdf9aaf5f98b8"
Jul 27 20:08:21.664630 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.664616    2562 scope.go:117] "RemoveContainer" containerID="0194ebff5882a6cacc97fe1f255634e632364267c9e464a1b225c9752b46cbbb"
Jul 27 20:08:21.672626 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.672605    2562 scope.go:117] "RemoveContainer" containerID="185437234b6a7a9d7a8a00f762371979d982d423f317e1b3bb1bdf9aaf5f98b8"
Jul 27 20:08:21.672911 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:08:21.672890    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"185437234b6a7a9d7a8a00f762371979d982d423f317e1b3bb1bdf9aaf5f98b8\": container with ID starting with 185437234b6a7a9d7a8a00f762371979d982d423f317e1b3bb1bdf9aaf5f98b8 not found: ID does not exist" containerID="185437234b6a7a9d7a8a00f762371979d982d423f317e1b3bb1bdf9aaf5f98b8"
Jul 27 20:08:21.672992 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.672919    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"185437234b6a7a9d7a8a00f762371979d982d423f317e1b3bb1bdf9aaf5f98b8"} err="failed to get container status \"185437234b6a7a9d7a8a00f762371979d982d423f317e1b3bb1bdf9aaf5f98b8\": rpc error: code = NotFound desc = could not find container \"185437234b6a7a9d7a8a00f762371979d982d423f317e1b3bb1bdf9aaf5f98b8\": container with ID starting with 185437234b6a7a9d7a8a00f762371979d982d423f317e1b3bb1bdf9aaf5f98b8 not found: ID does not exist"
Jul 27 20:08:21.672992 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.672937    2562 scope.go:117] "RemoveContainer" containerID="0194ebff5882a6cacc97fe1f255634e632364267c9e464a1b225c9752b46cbbb"
Jul 27 20:08:21.673163 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:08:21.673148    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0194ebff5882a6cacc97fe1f255634e632364267c9e464a1b225c9752b46cbbb\": container with ID starting with 0194ebff5882a6cacc97fe1f255634e632364267c9e464a1b225c9752b46cbbb not found: ID does not exist" containerID="0194ebff5882a6cacc97fe1f255634e632364267c9e464a1b225c9752b46cbbb"
Jul 27 20:08:21.673227 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.673170    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0194ebff5882a6cacc97fe1f255634e632364267c9e464a1b225c9752b46cbbb"} err="failed to get container status \"0194ebff5882a6cacc97fe1f255634e632364267c9e464a1b225c9752b46cbbb\": rpc error: code = NotFound desc = could not find container \"0194ebff5882a6cacc97fe1f255634e632364267c9e464a1b225c9752b46cbbb\": container with ID starting with 0194ebff5882a6cacc97fe1f255634e632364267c9e464a1b225c9752b46cbbb not found: ID does not exist"
Jul 27 20:08:21.681989 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.681960    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"]
Jul 27 20:08:21.687732 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.687715    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e3c0fa8a-openshift-default-945456b78-v6jr9"]
Jul 27 20:08:21.992816 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:21.992740    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b931e1d-bd27-480b-8444-f999433d79e9" path="/var/lib/kubelet/pods/3b931e1d-bd27-480b-8444-f999433d79e9/volumes"
Jul 27 20:08:26.659393 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:26.659359    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:32.725051 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:32.724965    2562 generic.go:356] "Generic (PLEG): container finished" podID="59dd910b-b601-4769-8a22-33f9583390a5" containerID="ee3097240eb21613550bf5845bfa1992f3b29531b9768edf59b19d03f8e460ce" exitCode=0
Jul 27 20:08:32.725051 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:32.725021    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f" event={"ID":"59dd910b-b601-4769-8a22-33f9583390a5","Type":"ContainerDied","Data":"ee3097240eb21613550bf5845bfa1992f3b29531b9768edf59b19d03f8e460ce"}
Jul 27 20:08:32.725509 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:32.725395    2562 scope.go:117] "RemoveContainer" containerID="ee3097240eb21613550bf5845bfa1992f3b29531b9768edf59b19d03f8e460ce"
Jul 27 20:08:32.726544 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:32.726514    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:08:32.726631 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:32.726591    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:08:32.726673 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:32.726630    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:08:33.735344 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:33.735296    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f" event={"ID":"59dd910b-b601-4769-8a22-33f9583390a5","Type":"ContainerStarted","Data":"6305c42237e6f54244ed511de7a60a3aad4b955d9974483a336031b3b8790a0b"}
Jul 27 20:08:33.890170 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:33.890125    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:33.890371 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:33.890308    2562 patch_prober.go:28] interesting pod/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 27 20:08:33.890441 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:33.890375    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f" podUID="59dd910b-b601-4769-8a22-33f9583390a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 27 20:08:34.638822 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:34.638782    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:34.887999 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:34.887951    2562 patch_prober.go:28] interesting pod/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 27 20:08:34.888532 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:34.888042    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f" podUID="59dd910b-b601-4769-8a22-33f9583390a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 27 20:08:35.888468 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:35.888418    2562 patch_prober.go:28] interesting pod/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 27 20:08:35.889122 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:35.889089    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f" podUID="59dd910b-b601-4769-8a22-33f9583390a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 27 20:08:36.888938 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:36.888894    2562 patch_prober.go:28] interesting pod/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 27 20:08:36.889448 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:36.888986    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f" podUID="59dd910b-b601-4769-8a22-33f9583390a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 27 20:08:37.891819 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:37.891791    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:37.892778 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:37.892756    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:08:54.449571 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.449526    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"]
Jul 27 20:08:54.450086 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.449757    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr" podUID="c2aeb3e9-1661-47da-9b58-38973a5beaa7" containerName="maas-api" containerID="cri-o://9f0e33c6ffa5174661b58ad48dd3380af58da0d162aac16d6cacc3785309998c" gracePeriod=30
Jul 27 20:08:54.729660 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.729633    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:54.820257 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.820225    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2aeb3e9-1661-47da-9b58-38973a5beaa7-openshift-service-ca\") pod \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\" (UID: \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\") "
Jul 27 20:08:54.820457 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.820272    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8css8\" (UniqueName: \"kubernetes.io/projected/c2aeb3e9-1661-47da-9b58-38973a5beaa7-kube-api-access-8css8\") pod \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\" (UID: \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\") "
Jul 27 20:08:54.820457 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.820344    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2aeb3e9-1661-47da-9b58-38973a5beaa7-maas-api-tls\") pod \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\" (UID: \"c2aeb3e9-1661-47da-9b58-38973a5beaa7\") "
Jul 27 20:08:54.820711 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.820685    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2aeb3e9-1661-47da-9b58-38973a5beaa7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2aeb3e9-1661-47da-9b58-38973a5beaa7" (UID: "c2aeb3e9-1661-47da-9b58-38973a5beaa7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:08:54.822677 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.822645    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2aeb3e9-1661-47da-9b58-38973a5beaa7-kube-api-access-8css8" (OuterVolumeSpecName: "kube-api-access-8css8") pod "c2aeb3e9-1661-47da-9b58-38973a5beaa7" (UID: "c2aeb3e9-1661-47da-9b58-38973a5beaa7"). InnerVolumeSpecName "kube-api-access-8css8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:08:54.822772 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.822668    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2aeb3e9-1661-47da-9b58-38973a5beaa7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2aeb3e9-1661-47da-9b58-38973a5beaa7" (UID: "c2aeb3e9-1661-47da-9b58-38973a5beaa7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:08:54.846662 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.846631    2562 generic.go:356] "Generic (PLEG): container finished" podID="c2aeb3e9-1661-47da-9b58-38973a5beaa7" containerID="9f0e33c6ffa5174661b58ad48dd3380af58da0d162aac16d6cacc3785309998c" exitCode=0
Jul 27 20:08:54.846816 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.846678    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"
Jul 27 20:08:54.846816 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.846687    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr" event={"ID":"c2aeb3e9-1661-47da-9b58-38973a5beaa7","Type":"ContainerDied","Data":"9f0e33c6ffa5174661b58ad48dd3380af58da0d162aac16d6cacc3785309998c"}
Jul 27 20:08:54.846816 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.846729    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr" event={"ID":"c2aeb3e9-1661-47da-9b58-38973a5beaa7","Type":"ContainerDied","Data":"c3af37e52e96167b6e7f0a8b1b9b5c4f6cdd9342d3e0dc044dab766bfbeb9ab7"}
Jul 27 20:08:54.846816 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.846749    2562 scope.go:117] "RemoveContainer" containerID="9f0e33c6ffa5174661b58ad48dd3380af58da0d162aac16d6cacc3785309998c"
Jul 27 20:08:54.855902 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.855884    2562 scope.go:117] "RemoveContainer" containerID="9f0e33c6ffa5174661b58ad48dd3380af58da0d162aac16d6cacc3785309998c"
Jul 27 20:08:54.856183 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:08:54.856164    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0e33c6ffa5174661b58ad48dd3380af58da0d162aac16d6cacc3785309998c\": container with ID starting with 9f0e33c6ffa5174661b58ad48dd3380af58da0d162aac16d6cacc3785309998c not found: ID does not exist" containerID="9f0e33c6ffa5174661b58ad48dd3380af58da0d162aac16d6cacc3785309998c"
Jul 27 20:08:54.856240 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.856193    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0e33c6ffa5174661b58ad48dd3380af58da0d162aac16d6cacc3785309998c"} err="failed to get container status \"9f0e33c6ffa5174661b58ad48dd3380af58da0d162aac16d6cacc3785309998c\": rpc error: code = NotFound desc = could not find container \"9f0e33c6ffa5174661b58ad48dd3380af58da0d162aac16d6cacc3785309998c\": container with ID starting with 9f0e33c6ffa5174661b58ad48dd3380af58da0d162aac16d6cacc3785309998c not found: ID does not exist"
Jul 27 20:08:54.868268 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.868237    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"]
Jul 27 20:08:54.874464 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.874440    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-269943c6-85df6c4878-5pjtr"]
Jul 27 20:08:54.922040 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.922006    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2aeb3e9-1661-47da-9b58-38973a5beaa7-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:54.922040 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.922038    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8css8\" (UniqueName: \"kubernetes.io/projected/c2aeb3e9-1661-47da-9b58-38973a5beaa7-kube-api-access-8css8\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:54.922040 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:54.922049    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2aeb3e9-1661-47da-9b58-38973a5beaa7-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:08:55.992109 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:55.992073    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2aeb3e9-1661-47da-9b58-38973a5beaa7" path="/var/lib/kubelet/pods/c2aeb3e9-1661-47da-9b58-38973a5beaa7/volumes"
Jul 27 20:08:59.069643 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:59.069604    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"]
Jul 27 20:08:59.070119 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:08:59.069865    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f" podUID="59dd910b-b601-4769-8a22-33f9583390a5" containerName="istio-proxy" containerID="cri-o://6305c42237e6f54244ed511de7a60a3aad4b955d9974483a336031b3b8790a0b" gracePeriod=30
Jul 27 20:09:04.337122 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.337059    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:09:04.514905 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.514873    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dd910b-b601-4769-8a22-33f9583390a5-istiod-ca-cert\") pod \"59dd910b-b601-4769-8a22-33f9583390a5\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") "
Jul 27 20:09:04.514905 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.514912    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dd910b-b601-4769-8a22-33f9583390a5-istio-podinfo\") pod \"59dd910b-b601-4769-8a22-33f9583390a5\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") "
Jul 27 20:09:04.515148 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.514945    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-workload-socket\") pod \"59dd910b-b601-4769-8a22-33f9583390a5\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") "
Jul 27 20:09:04.515148 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.515026    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-workload-certs\") pod \"59dd910b-b601-4769-8a22-33f9583390a5\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") "
Jul 27 20:09:04.515148 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.515064    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-istio-envoy\") pod \"59dd910b-b601-4769-8a22-33f9583390a5\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") "
Jul 27 20:09:04.515148 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.515130    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-istio-data\") pod \"59dd910b-b601-4769-8a22-33f9583390a5\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") "
Jul 27 20:09:04.515148 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.515149    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c6w7\" (UniqueName: \"kubernetes.io/projected/59dd910b-b601-4769-8a22-33f9583390a5-kube-api-access-8c6w7\") pod \"59dd910b-b601-4769-8a22-33f9583390a5\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") "
Jul 27 20:09:04.515410 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.515186    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-credential-socket\") pod \"59dd910b-b601-4769-8a22-33f9583390a5\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") "
Jul 27 20:09:04.515410 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.515198    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "59dd910b-b601-4769-8a22-33f9583390a5" (UID: "59dd910b-b601-4769-8a22-33f9583390a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:09:04.515410 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.515228    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59dd910b-b601-4769-8a22-33f9583390a5-istio-token\") pod \"59dd910b-b601-4769-8a22-33f9583390a5\" (UID: \"59dd910b-b601-4769-8a22-33f9583390a5\") "
Jul 27 20:09:04.515410 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.515320    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59dd910b-b601-4769-8a22-33f9583390a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "59dd910b-b601-4769-8a22-33f9583390a5" (UID: "59dd910b-b601-4769-8a22-33f9583390a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:09:04.515410 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.515341    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "59dd910b-b601-4769-8a22-33f9583390a5" (UID: "59dd910b-b601-4769-8a22-33f9583390a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:09:04.515701 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.515608    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "59dd910b-b601-4769-8a22-33f9583390a5" (UID: "59dd910b-b601-4769-8a22-33f9583390a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:09:04.515750 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.515694    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-workload-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:09:04.515750 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.515715    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-workload-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:09:04.515750 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.515729    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dd910b-b601-4769-8a22-33f9583390a5-istiod-ca-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:09:04.516107 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.516082    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "59dd910b-b601-4769-8a22-33f9583390a5" (UID: "59dd910b-b601-4769-8a22-33f9583390a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:09:04.517345 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.517319    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "59dd910b-b601-4769-8a22-33f9583390a5" (UID: "59dd910b-b601-4769-8a22-33f9583390a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:09:04.517555 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.517534    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dd910b-b601-4769-8a22-33f9583390a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "59dd910b-b601-4769-8a22-33f9583390a5" (UID: "59dd910b-b601-4769-8a22-33f9583390a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:09:04.517881 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.517853    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59dd910b-b601-4769-8a22-33f9583390a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "59dd910b-b601-4769-8a22-33f9583390a5" (UID: "59dd910b-b601-4769-8a22-33f9583390a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:09:04.517881 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.517858    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dd910b-b601-4769-8a22-33f9583390a5-kube-api-access-8c6w7" (OuterVolumeSpecName: "kube-api-access-8c6w7") pod "59dd910b-b601-4769-8a22-33f9583390a5" (UID: "59dd910b-b601-4769-8a22-33f9583390a5"). InnerVolumeSpecName "kube-api-access-8c6w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:09:04.616892 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.616843    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dd910b-b601-4769-8a22-33f9583390a5-istio-podinfo\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:09:04.616892 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.616866    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-istio-envoy\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:09:04.616892 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.616875    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-istio-data\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:09:04.616892 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.616884    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c6w7\" (UniqueName: \"kubernetes.io/projected/59dd910b-b601-4769-8a22-33f9583390a5-kube-api-access-8c6w7\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:09:04.616892 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.616893    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dd910b-b601-4769-8a22-33f9583390a5-credential-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:09:04.617173 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.616903    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59dd910b-b601-4769-8a22-33f9583390a5-istio-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:09:04.903066 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.902984    2562 generic.go:356] "Generic (PLEG): container finished" podID="59dd910b-b601-4769-8a22-33f9583390a5" containerID="6305c42237e6f54244ed511de7a60a3aad4b955d9974483a336031b3b8790a0b" exitCode=0
Jul 27 20:09:04.903066 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.903034    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"
Jul 27 20:09:04.903066 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.903059    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f" event={"ID":"59dd910b-b601-4769-8a22-33f9583390a5","Type":"ContainerDied","Data":"6305c42237e6f54244ed511de7a60a3aad4b955d9974483a336031b3b8790a0b"}
Jul 27 20:09:04.903310 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.903111    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f" event={"ID":"59dd910b-b601-4769-8a22-33f9583390a5","Type":"ContainerDied","Data":"2320e693497488ff04453dfd853328e8ee23fe59b9b0547666e56ad296659194"}
Jul 27 20:09:04.903310 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.903134    2562 scope.go:117] "RemoveContainer" containerID="6305c42237e6f54244ed511de7a60a3aad4b955d9974483a336031b3b8790a0b"
Jul 27 20:09:04.912694 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.912678    2562 scope.go:117] "RemoveContainer" containerID="ee3097240eb21613550bf5845bfa1992f3b29531b9768edf59b19d03f8e460ce"
Jul 27 20:09:04.920571 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.920556    2562 scope.go:117] "RemoveContainer" containerID="6305c42237e6f54244ed511de7a60a3aad4b955d9974483a336031b3b8790a0b"
Jul 27 20:09:04.920825 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:09:04.920807    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6305c42237e6f54244ed511de7a60a3aad4b955d9974483a336031b3b8790a0b\": container with ID starting with 6305c42237e6f54244ed511de7a60a3aad4b955d9974483a336031b3b8790a0b not found: ID does not exist" containerID="6305c42237e6f54244ed511de7a60a3aad4b955d9974483a336031b3b8790a0b"
Jul 27 20:09:04.920894 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.920833    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6305c42237e6f54244ed511de7a60a3aad4b955d9974483a336031b3b8790a0b"} err="failed to get container status \"6305c42237e6f54244ed511de7a60a3aad4b955d9974483a336031b3b8790a0b\": rpc error: code = NotFound desc = could not find container \"6305c42237e6f54244ed511de7a60a3aad4b955d9974483a336031b3b8790a0b\": container with ID starting with 6305c42237e6f54244ed511de7a60a3aad4b955d9974483a336031b3b8790a0b not found: ID does not exist"
Jul 27 20:09:04.920894 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.920849    2562 scope.go:117] "RemoveContainer" containerID="ee3097240eb21613550bf5845bfa1992f3b29531b9768edf59b19d03f8e460ce"
Jul 27 20:09:04.921056 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:09:04.921038    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee3097240eb21613550bf5845bfa1992f3b29531b9768edf59b19d03f8e460ce\": container with ID starting with ee3097240eb21613550bf5845bfa1992f3b29531b9768edf59b19d03f8e460ce not found: ID does not exist" containerID="ee3097240eb21613550bf5845bfa1992f3b29531b9768edf59b19d03f8e460ce"
Jul 27 20:09:04.921093 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.921059    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee3097240eb21613550bf5845bfa1992f3b29531b9768edf59b19d03f8e460ce"} err="failed to get container status \"ee3097240eb21613550bf5845bfa1992f3b29531b9768edf59b19d03f8e460ce\": rpc error: code = NotFound desc = could not find container \"ee3097240eb21613550bf5845bfa1992f3b29531b9768edf59b19d03f8e460ce\": container with ID starting with ee3097240eb21613550bf5845bfa1992f3b29531b9768edf59b19d03f8e460ce not found: ID does not exist"
Jul 27 20:09:04.924629 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.924608    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"]
Jul 27 20:09:04.928674 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:04.928654    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-269943c6-openshift-default-cf9c5c4b4-x859f"]
Jul 27 20:09:05.991367 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:09:05.991332    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59dd910b-b601-4769-8a22-33f9583390a5" path="/var/lib/kubelet/pods/59dd910b-b601-4769-8a22-33f9583390a5/volumes"
Jul 27 20:10:39.518283 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518251    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"]
Jul 27 20:10:39.518767 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518685    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b931e1d-bd27-480b-8444-f999433d79e9" containerName="istio-proxy"
Jul 27 20:10:39.518767 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518697    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b931e1d-bd27-480b-8444-f999433d79e9" containerName="istio-proxy"
Jul 27 20:10:39.518767 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518709    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59dd910b-b601-4769-8a22-33f9583390a5" containerName="istio-proxy"
Jul 27 20:10:39.518767 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518715    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="59dd910b-b601-4769-8a22-33f9583390a5" containerName="istio-proxy"
Jul 27 20:10:39.518767 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518726    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59dd910b-b601-4769-8a22-33f9583390a5" containerName="istio-proxy"
Jul 27 20:10:39.518767 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518732    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="59dd910b-b601-4769-8a22-33f9583390a5" containerName="istio-proxy"
Jul 27 20:10:39.518767 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518741    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b931e1d-bd27-480b-8444-f999433d79e9" containerName="istio-proxy"
Jul 27 20:10:39.518767 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518745    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b931e1d-bd27-480b-8444-f999433d79e9" containerName="istio-proxy"
Jul 27 20:10:39.518767 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518752    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2aeb3e9-1661-47da-9b58-38973a5beaa7" containerName="maas-api"
Jul 27 20:10:39.518767 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518757    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2aeb3e9-1661-47da-9b58-38973a5beaa7" containerName="maas-api"
Jul 27 20:10:39.519072 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518818    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b931e1d-bd27-480b-8444-f999433d79e9" containerName="istio-proxy"
Jul 27 20:10:39.519072 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518825    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2aeb3e9-1661-47da-9b58-38973a5beaa7" containerName="maas-api"
Jul 27 20:10:39.519072 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518832    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b931e1d-bd27-480b-8444-f999433d79e9" containerName="istio-proxy"
Jul 27 20:10:39.519072 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.518838    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="59dd910b-b601-4769-8a22-33f9583390a5" containerName="istio-proxy"
Jul 27 20:10:39.522193 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.522175    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.524364 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.524340    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-452dddb4-openshift-default-dockercfg-sb92m\""
Jul 27 20:10:39.584883 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.584849    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"]
Jul 27 20:10:39.632605 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.632571    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5npth\" (UniqueName: \"kubernetes.io/projected/b65f5bde-0c0b-4774-8a4d-c897a04d530a-kube-api-access-5npth\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.632780 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.632614    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-token\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.632780 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.632668    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istiod-ca-cert\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.632780 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.632695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-workload-certs\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.632886 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.632780    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-envoy\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.632886 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.632828    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-podinfo\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.632886 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.632863    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-data\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.632886 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.632879    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-credential-socket\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.633023 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.632907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-workload-socket\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.733790 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.733752    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-workload-certs\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.733995 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.733805    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-envoy\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.733995 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.733835    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-podinfo\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.733995 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.733867    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-data\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.733995 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.733890    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-credential-socket\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.733995 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.733925    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-workload-socket\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.733995 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.733974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5npth\" (UniqueName: \"kubernetes.io/projected/b65f5bde-0c0b-4774-8a4d-c897a04d530a-kube-api-access-5npth\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.734332 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.734005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-token\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.734332 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.734046    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istiod-ca-cert\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.734332 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.734250    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-workload-certs\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.734584 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.734331    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-data\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.734584 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.734350    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-workload-socket\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.734668 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.734627    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-credential-socket\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.735111 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.735087    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istiod-ca-cert\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.736317 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.736298    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-envoy\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.736430 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.736412    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-podinfo\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.743741 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.743720    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-token\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.743951 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.743928    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5npth\" (UniqueName: \"kubernetes.io/projected/b65f5bde-0c0b-4774-8a4d-c897a04d530a-kube-api-access-5npth\") pod \"e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") " pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.833056 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.832966    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:39.999237 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:10:39.999204    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65f5bde_0c0b_4774_8a4d_c897a04d530a.slice/crio-dcb34b3f8486cb829da1cf5428e9b724fa1a2e17e377bb764024cc8c08e67b52 WatchSource:0}: Error finding container dcb34b3f8486cb829da1cf5428e9b724fa1a2e17e377bb764024cc8c08e67b52: Status 404 returned error can't find the container with id dcb34b3f8486cb829da1cf5428e9b724fa1a2e17e377bb764024cc8c08e67b52
Jul 27 20:10:39.999434 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:39.999415    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"]
Jul 27 20:10:40.001283 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:40.001237    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:10:40.001363 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:40.001318    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:10:40.001363 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:40.001346    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:10:40.397128 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:40.397093    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r" event={"ID":"b65f5bde-0c0b-4774-8a4d-c897a04d530a","Type":"ContainerStarted","Data":"b15809f68a7ac3374fe2f942c5b78741d9a5197f68b43603404ccc7915a4b314"}
Jul 27 20:10:40.397298 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:40.397145    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r" event={"ID":"b65f5bde-0c0b-4774-8a4d-c897a04d530a","Type":"ContainerStarted","Data":"dcb34b3f8486cb829da1cf5428e9b724fa1a2e17e377bb764024cc8c08e67b52"}
Jul 27 20:10:40.424825 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:40.424758    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r" podStartSLOduration=1.4247421679999999 podStartE2EDuration="1.424742168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:10:40.421662168 +0000 UTC m=+1854.973578587" watchObservedRunningTime="2026-07-27 20:10:40.424742168 +0000 UTC m=+1854.976658656"
Jul 27 20:10:40.833668 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:40.833565    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:41.838471 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:41.838440    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:42.407363 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.407328    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:42.408335 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.408317    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:10:42.467045 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.467006    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"]
Jul 27 20:10:42.467949 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.467916    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="59dd910b-b601-4769-8a22-33f9583390a5" containerName="istio-proxy"
Jul 27 20:10:42.471473 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.471452    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:10:42.473682 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.473652    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-452dddb4-serving-cert\""
Jul 27 20:10:42.484449 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.484139    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"]
Jul 27 20:10:42.562304 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.562273    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf5896e-21d9-45c0-88bd-5989730185dc-maas-api-tls\") pod \"maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8\" (UID: \"9bf5896e-21d9-45c0-88bd-5989730185dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:10:42.562524 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.562337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf5896e-21d9-45c0-88bd-5989730185dc-openshift-service-ca\") pod \"maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8\" (UID: \"9bf5896e-21d9-45c0-88bd-5989730185dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:10:42.562524 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.562435    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2g7z\" (UniqueName: \"kubernetes.io/projected/9bf5896e-21d9-45c0-88bd-5989730185dc-kube-api-access-c2g7z\") pod \"maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8\" (UID: \"9bf5896e-21d9-45c0-88bd-5989730185dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:10:42.663980 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.663879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf5896e-21d9-45c0-88bd-5989730185dc-openshift-service-ca\") pod \"maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8\" (UID: \"9bf5896e-21d9-45c0-88bd-5989730185dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:10:42.663980 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.663937    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2g7z\" (UniqueName: \"kubernetes.io/projected/9bf5896e-21d9-45c0-88bd-5989730185dc-kube-api-access-c2g7z\") pod \"maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8\" (UID: \"9bf5896e-21d9-45c0-88bd-5989730185dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:10:42.664203 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.664008    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf5896e-21d9-45c0-88bd-5989730185dc-maas-api-tls\") pod \"maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8\" (UID: \"9bf5896e-21d9-45c0-88bd-5989730185dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:10:42.664203 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:10:42.664177    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-serving-cert: secret "maas-api-e2e-mt-452dddb4-serving-cert" not found
Jul 27 20:10:42.664284 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:10:42.664262    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf5896e-21d9-45c0-88bd-5989730185dc-maas-api-tls podName:9bf5896e-21d9-45c0-88bd-5989730185dc nodeName:}" failed. No retries permitted until 2026-07-27 20:10:43.164241145 +0000 UTC m=+1857.716157547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9bf5896e-21d9-45c0-88bd-5989730185dc-maas-api-tls") pod "maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8" (UID: "9bf5896e-21d9-45c0-88bd-5989730185dc") : secret "maas-api-e2e-mt-452dddb4-serving-cert" not found
Jul 27 20:10:42.664626 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.664608    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf5896e-21d9-45c0-88bd-5989730185dc-openshift-service-ca\") pod \"maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8\" (UID: \"9bf5896e-21d9-45c0-88bd-5989730185dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:10:42.677828 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:42.677794    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2g7z\" (UniqueName: \"kubernetes.io/projected/9bf5896e-21d9-45c0-88bd-5989730185dc-kube-api-access-c2g7z\") pod \"maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8\" (UID: \"9bf5896e-21d9-45c0-88bd-5989730185dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:10:43.169526 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:43.169487    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf5896e-21d9-45c0-88bd-5989730185dc-maas-api-tls\") pod \"maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8\" (UID: \"9bf5896e-21d9-45c0-88bd-5989730185dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:10:43.172055 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:43.172031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf5896e-21d9-45c0-88bd-5989730185dc-maas-api-tls\") pod \"maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8\" (UID: \"9bf5896e-21d9-45c0-88bd-5989730185dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:10:43.387756 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:43.387705    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:10:43.539754 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:43.539725    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"]
Jul 27 20:10:43.542199 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:10:43.542173    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf5896e_21d9_45c0_88bd_5989730185dc.slice/crio-780392b441087aaadb22111f52d12aa49c7954b6a21397796b2824c5a0a17ec8 WatchSource:0}: Error finding container 780392b441087aaadb22111f52d12aa49c7954b6a21397796b2824c5a0a17ec8: Status 404 returned error can't find the container with id 780392b441087aaadb22111f52d12aa49c7954b6a21397796b2824c5a0a17ec8
Jul 27 20:10:44.426649 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:44.426612    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8" event={"ID":"9bf5896e-21d9-45c0-88bd-5989730185dc","Type":"ContainerStarted","Data":"4c2c0eac169f1940a8cced09070c5dc448797455b6769c74c46c55a514b06e92"}
Jul 27 20:10:44.427042 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:44.426692    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8" event={"ID":"9bf5896e-21d9-45c0-88bd-5989730185dc","Type":"ContainerStarted","Data":"780392b441087aaadb22111f52d12aa49c7954b6a21397796b2824c5a0a17ec8"}
Jul 27 20:10:44.427042 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:44.426872    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:10:44.447619 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:44.447568    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8" podStartSLOduration=2.01727894 podStartE2EDuration="2.447554314s" totalImagesPullingTime="430.275374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:10:44.445458277 +0000 UTC m=+1858.997374695" watchObservedRunningTime="2026-07-27 20:10:44.447554314 +0000 UTC m=+1858.999470736"
Jul 27 20:10:50.437523 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:50.437494    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:10:58.449676 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:58.449620    2562 patch_prober.go:28] interesting pod/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 20:10:58.450160 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:58.449697    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r" podUID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 20:10:59.514852 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:59.514752    2562 generic.go:356] "Generic (PLEG): container finished" podID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerID="b15809f68a7ac3374fe2f942c5b78741d9a5197f68b43603404ccc7915a4b314" exitCode=0
Jul 27 20:10:59.514852 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:59.514826    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r" event={"ID":"b65f5bde-0c0b-4774-8a4d-c897a04d530a","Type":"ContainerDied","Data":"b15809f68a7ac3374fe2f942c5b78741d9a5197f68b43603404ccc7915a4b314"}
Jul 27 20:10:59.515362 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:59.515224    2562 scope.go:117] "RemoveContainer" containerID="b15809f68a7ac3374fe2f942c5b78741d9a5197f68b43603404ccc7915a4b314"
Jul 27 20:10:59.516245 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:59.516214    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:10:59.516345 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:59.516309    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:10:59.516410 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:10:59.516369    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:11:00.521713 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:00.521671    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r" event={"ID":"b65f5bde-0c0b-4774-8a4d-c897a04d530a","Type":"ContainerStarted","Data":"3a9ca1d3052e9cb91f39a7224f8516c49bea3be4aa87580ce7b07347183dd47d"}
Jul 27 20:11:00.833675 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:00.833581    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:11:00.833907 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:00.833884    2562 patch_prober.go:28] interesting pod/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 27 20:11:00.833982 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:00.833964    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r" podUID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 27 20:11:01.834347 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:01.834305    2562 patch_prober.go:28] interesting pod/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 27 20:11:01.834774 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:01.834380    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r" podUID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 27 20:11:02.833798 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:02.833756    2562 patch_prober.go:28] interesting pod/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 27 20:11:02.833984 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:02.833816    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r" podUID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 27 20:11:03.834000 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:03.833966    2562 patch_prober.go:28] interesting pod/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 27 20:11:03.834410 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:03.834027    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r" podUID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 27 20:11:04.836832 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:04.836801    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:11:04.837312 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:04.837188    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:11:04.837892 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:04.837870    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:11:04.909739 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:04.909699    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"]
Jul 27 20:11:04.910209 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:04.910181    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8" podUID="9bf5896e-21d9-45c0-88bd-5989730185dc" containerName="maas-api" containerID="cri-o://4c2c0eac169f1940a8cced09070c5dc448797455b6769c74c46c55a514b06e92" gracePeriod=30
Jul 27 20:11:05.174210 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.174182    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:11:05.314455 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.314415    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf5896e-21d9-45c0-88bd-5989730185dc-maas-api-tls\") pod \"9bf5896e-21d9-45c0-88bd-5989730185dc\" (UID: \"9bf5896e-21d9-45c0-88bd-5989730185dc\") "
Jul 27 20:11:05.314455 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.314453    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf5896e-21d9-45c0-88bd-5989730185dc-openshift-service-ca\") pod \"9bf5896e-21d9-45c0-88bd-5989730185dc\" (UID: \"9bf5896e-21d9-45c0-88bd-5989730185dc\") "
Jul 27 20:11:05.314737 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.314524    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2g7z\" (UniqueName: \"kubernetes.io/projected/9bf5896e-21d9-45c0-88bd-5989730185dc-kube-api-access-c2g7z\") pod \"9bf5896e-21d9-45c0-88bd-5989730185dc\" (UID: \"9bf5896e-21d9-45c0-88bd-5989730185dc\") "
Jul 27 20:11:05.314803 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.314772    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf5896e-21d9-45c0-88bd-5989730185dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9bf5896e-21d9-45c0-88bd-5989730185dc" (UID: "9bf5896e-21d9-45c0-88bd-5989730185dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:11:05.316769 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.316735    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf5896e-21d9-45c0-88bd-5989730185dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9bf5896e-21d9-45c0-88bd-5989730185dc" (UID: "9bf5896e-21d9-45c0-88bd-5989730185dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:11:05.316886 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.316815    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf5896e-21d9-45c0-88bd-5989730185dc-kube-api-access-c2g7z" (OuterVolumeSpecName: "kube-api-access-c2g7z") pod "9bf5896e-21d9-45c0-88bd-5989730185dc" (UID: "9bf5896e-21d9-45c0-88bd-5989730185dc"). InnerVolumeSpecName "kube-api-access-c2g7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:11:05.416098 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.416001    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf5896e-21d9-45c0-88bd-5989730185dc-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:11:05.416098 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.416036    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf5896e-21d9-45c0-88bd-5989730185dc-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:11:05.416098 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.416048    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2g7z\" (UniqueName: \"kubernetes.io/projected/9bf5896e-21d9-45c0-88bd-5989730185dc-kube-api-access-c2g7z\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:11:05.550868 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.550829    2562 generic.go:356] "Generic (PLEG): container finished" podID="9bf5896e-21d9-45c0-88bd-5989730185dc" containerID="4c2c0eac169f1940a8cced09070c5dc448797455b6769c74c46c55a514b06e92" exitCode=0
Jul 27 20:11:05.551065 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.550897    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"
Jul 27 20:11:05.551065 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.550914    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8" event={"ID":"9bf5896e-21d9-45c0-88bd-5989730185dc","Type":"ContainerDied","Data":"4c2c0eac169f1940a8cced09070c5dc448797455b6769c74c46c55a514b06e92"}
Jul 27 20:11:05.551065 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.550996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8" event={"ID":"9bf5896e-21d9-45c0-88bd-5989730185dc","Type":"ContainerDied","Data":"780392b441087aaadb22111f52d12aa49c7954b6a21397796b2824c5a0a17ec8"}
Jul 27 20:11:05.551065 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.551018    2562 scope.go:117] "RemoveContainer" containerID="4c2c0eac169f1940a8cced09070c5dc448797455b6769c74c46c55a514b06e92"
Jul 27 20:11:05.560311 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.560293    2562 scope.go:117] "RemoveContainer" containerID="4c2c0eac169f1940a8cced09070c5dc448797455b6769c74c46c55a514b06e92"
Jul 27 20:11:05.560631 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:11:05.560610    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2c0eac169f1940a8cced09070c5dc448797455b6769c74c46c55a514b06e92\": container with ID starting with 4c2c0eac169f1940a8cced09070c5dc448797455b6769c74c46c55a514b06e92 not found: ID does not exist" containerID="4c2c0eac169f1940a8cced09070c5dc448797455b6769c74c46c55a514b06e92"
Jul 27 20:11:05.560709 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.560638    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2c0eac169f1940a8cced09070c5dc448797455b6769c74c46c55a514b06e92"} err="failed to get container status \"4c2c0eac169f1940a8cced09070c5dc448797455b6769c74c46c55a514b06e92\": rpc error: code = NotFound desc = could not find container \"4c2c0eac169f1940a8cced09070c5dc448797455b6769c74c46c55a514b06e92\": container with ID starting with 4c2c0eac169f1940a8cced09070c5dc448797455b6769c74c46c55a514b06e92 not found: ID does not exist"
Jul 27 20:11:05.572691 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.572664    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"]
Jul 27 20:11:05.578210 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.578178    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-452dddb4-64b65f948f-xp6x8"]
Jul 27 20:11:05.992109 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:05.992077    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bf5896e-21d9-45c0-88bd-5989730185dc" path="/var/lib/kubelet/pods/9bf5896e-21d9-45c0-88bd-5989730185dc/volumes"
Jul 27 20:11:16.252686 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:16.252598    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"]
Jul 27 20:11:16.253256 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:16.252827    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r" podUID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerName="istio-proxy" containerID="cri-o://3a9ca1d3052e9cb91f39a7224f8516c49bea3be4aa87580ce7b07347183dd47d" gracePeriod=30
Jul 27 20:11:19.839806 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:19.839766    2562 patch_prober.go:28] interesting pod/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 20:11:19.840187 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:19.839840    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r" podUID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 20:11:21.522395 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.522331    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:11:21.558910 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.558883    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-workload-certs\") pod \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") "
Jul 27 20:11:21.558910 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.558921    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-data\") pod \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") "
Jul 27 20:11:21.559146 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.558976    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-envoy\") pod \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") "
Jul 27 20:11:21.559146 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559010    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-podinfo\") pod \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") "
Jul 27 20:11:21.559146 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559061    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-credential-socket\") pod \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") "
Jul 27 20:11:21.559146 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559089    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istiod-ca-cert\") pod \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") "
Jul 27 20:11:21.559146 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559118    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-token\") pod \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") "
Jul 27 20:11:21.559402 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559195    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b65f5bde-0c0b-4774-8a4d-c897a04d530a" (UID: "b65f5bde-0c0b-4774-8a4d-c897a04d530a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:11:21.559402 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559232    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5npth\" (UniqueName: \"kubernetes.io/projected/b65f5bde-0c0b-4774-8a4d-c897a04d530a-kube-api-access-5npth\") pod \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") "
Jul 27 20:11:21.559402 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559309    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-workload-socket\") pod \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\" (UID: \"b65f5bde-0c0b-4774-8a4d-c897a04d530a\") "
Jul 27 20:11:21.559588 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559401    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b65f5bde-0c0b-4774-8a4d-c897a04d530a" (UID: "b65f5bde-0c0b-4774-8a4d-c897a04d530a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:11:21.559588 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559521    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b65f5bde-0c0b-4774-8a4d-c897a04d530a" (UID: "b65f5bde-0c0b-4774-8a4d-c897a04d530a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:11:21.559770 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559717    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b65f5bde-0c0b-4774-8a4d-c897a04d530a" (UID: "b65f5bde-0c0b-4774-8a4d-c897a04d530a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:11:21.559945 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559835    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-workload-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:11:21.559945 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559857    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-workload-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:11:21.559945 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559872    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-credential-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:11:21.559945 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559886    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istiod-ca-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:11:21.559945 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.559903    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-data" (OuterVolumeSpecName: "istio-data") pod "b65f5bde-0c0b-4774-8a4d-c897a04d530a" (UID: "b65f5bde-0c0b-4774-8a4d-c897a04d530a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:11:21.561505 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.561454    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b65f5bde-0c0b-4774-8a4d-c897a04d530a" (UID: "b65f5bde-0c0b-4774-8a4d-c897a04d530a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:11:21.561674 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.561651    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65f5bde-0c0b-4774-8a4d-c897a04d530a-kube-api-access-5npth" (OuterVolumeSpecName: "kube-api-access-5npth") pod "b65f5bde-0c0b-4774-8a4d-c897a04d530a" (UID: "b65f5bde-0c0b-4774-8a4d-c897a04d530a"). InnerVolumeSpecName "kube-api-access-5npth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:11:21.561832 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.561811    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b65f5bde-0c0b-4774-8a4d-c897a04d530a" (UID: "b65f5bde-0c0b-4774-8a4d-c897a04d530a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:11:21.562021 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.562005    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-token" (OuterVolumeSpecName: "istio-token") pod "b65f5bde-0c0b-4774-8a4d-c897a04d530a" (UID: "b65f5bde-0c0b-4774-8a4d-c897a04d530a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:11:21.637382 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.637341    2562 generic.go:356] "Generic (PLEG): container finished" podID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerID="3a9ca1d3052e9cb91f39a7224f8516c49bea3be4aa87580ce7b07347183dd47d" exitCode=0
Jul 27 20:11:21.637565 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.637390    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"
Jul 27 20:11:21.637565 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.637405    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r" event={"ID":"b65f5bde-0c0b-4774-8a4d-c897a04d530a","Type":"ContainerDied","Data":"3a9ca1d3052e9cb91f39a7224f8516c49bea3be4aa87580ce7b07347183dd47d"}
Jul 27 20:11:21.637565 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.637460    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r" event={"ID":"b65f5bde-0c0b-4774-8a4d-c897a04d530a","Type":"ContainerDied","Data":"dcb34b3f8486cb829da1cf5428e9b724fa1a2e17e377bb764024cc8c08e67b52"}
Jul 27 20:11:21.637565 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.637509    2562 scope.go:117] "RemoveContainer" containerID="3a9ca1d3052e9cb91f39a7224f8516c49bea3be4aa87580ce7b07347183dd47d"
Jul 27 20:11:21.647174 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.647154    2562 scope.go:117] "RemoveContainer" containerID="b15809f68a7ac3374fe2f942c5b78741d9a5197f68b43603404ccc7915a4b314"
Jul 27 20:11:21.655053 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.655032    2562 scope.go:117] "RemoveContainer" containerID="3a9ca1d3052e9cb91f39a7224f8516c49bea3be4aa87580ce7b07347183dd47d"
Jul 27 20:11:21.655316 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:11:21.655297    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9ca1d3052e9cb91f39a7224f8516c49bea3be4aa87580ce7b07347183dd47d\": container with ID starting with 3a9ca1d3052e9cb91f39a7224f8516c49bea3be4aa87580ce7b07347183dd47d not found: ID does not exist" containerID="3a9ca1d3052e9cb91f39a7224f8516c49bea3be4aa87580ce7b07347183dd47d"
Jul 27 20:11:21.655377 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.655325    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9ca1d3052e9cb91f39a7224f8516c49bea3be4aa87580ce7b07347183dd47d"} err="failed to get container status \"3a9ca1d3052e9cb91f39a7224f8516c49bea3be4aa87580ce7b07347183dd47d\": rpc error: code = NotFound desc = could not find container \"3a9ca1d3052e9cb91f39a7224f8516c49bea3be4aa87580ce7b07347183dd47d\": container with ID starting with 3a9ca1d3052e9cb91f39a7224f8516c49bea3be4aa87580ce7b07347183dd47d not found: ID does not exist"
Jul 27 20:11:21.655377 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.655343    2562 scope.go:117] "RemoveContainer" containerID="b15809f68a7ac3374fe2f942c5b78741d9a5197f68b43603404ccc7915a4b314"
Jul 27 20:11:21.655592 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:11:21.655573    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b15809f68a7ac3374fe2f942c5b78741d9a5197f68b43603404ccc7915a4b314\": container with ID starting with b15809f68a7ac3374fe2f942c5b78741d9a5197f68b43603404ccc7915a4b314 not found: ID does not exist" containerID="b15809f68a7ac3374fe2f942c5b78741d9a5197f68b43603404ccc7915a4b314"
Jul 27 20:11:21.655654 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.655601    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15809f68a7ac3374fe2f942c5b78741d9a5197f68b43603404ccc7915a4b314"} err="failed to get container status \"b15809f68a7ac3374fe2f942c5b78741d9a5197f68b43603404ccc7915a4b314\": rpc error: code = NotFound desc = could not find container \"b15809f68a7ac3374fe2f942c5b78741d9a5197f68b43603404ccc7915a4b314\": container with ID starting with b15809f68a7ac3374fe2f942c5b78741d9a5197f68b43603404ccc7915a4b314 not found: ID does not exist"
Jul 27 20:11:21.661110 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.661088    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-data\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:11:21.661191 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.661114    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-envoy\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:11:21.661191 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.661127    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-podinfo\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:11:21.661191 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.661139    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65f5bde-0c0b-4774-8a4d-c897a04d530a-istio-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:11:21.661191 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.661151    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5npth\" (UniqueName: \"kubernetes.io/projected/b65f5bde-0c0b-4774-8a4d-c897a04d530a-kube-api-access-5npth\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:11:21.665257 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.665233    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"]
Jul 27 20:11:21.678859 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.678833    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-452dddb4-openshift-default-8cbdd8dfb-wqn2r"]
Jul 27 20:11:21.991833 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:21.991802    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" path="/var/lib/kubelet/pods/b65f5bde-0c0b-4774-8a4d-c897a04d530a/volumes"
Jul 27 20:11:36.443654 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.443623    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"]
Jul 27 20:11:36.444125 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.444027    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerName="istio-proxy"
Jul 27 20:11:36.444125 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.444038    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerName="istio-proxy"
Jul 27 20:11:36.444125 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.444059    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bf5896e-21d9-45c0-88bd-5989730185dc" containerName="maas-api"
Jul 27 20:11:36.444125 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.444064    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bf5896e-21d9-45c0-88bd-5989730185dc" containerName="maas-api"
Jul 27 20:11:36.444343 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.444132    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerName="istio-proxy"
Jul 27 20:11:36.444343 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.444142    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerName="istio-proxy"
Jul 27 20:11:36.444343 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.444149    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bf5896e-21d9-45c0-88bd-5989730185dc" containerName="maas-api"
Jul 27 20:11:36.449013 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.448996    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.451146 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.451126    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-25d7f4d7-openshift-default-dockercfg-lhbp8\""
Jul 27 20:11:36.458891 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.458869    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"]
Jul 27 20:11:36.606762 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.606719    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-credential-socket\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.606762 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.606768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-istio-envoy\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.606978 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.606836    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-istio-data\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.606978 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.606871    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37aa2b03-ecca-44a1-b782-be384795d961-istio-podinfo\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.606978 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.606903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-workload-certs\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.606978 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.606955    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65whr\" (UniqueName: \"kubernetes.io/projected/37aa2b03-ecca-44a1-b782-be384795d961-kube-api-access-65whr\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.607115 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.606989    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37aa2b03-ecca-44a1-b782-be384795d961-istio-token\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.607115 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.607006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-workload-socket\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.607115 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.607040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37aa2b03-ecca-44a1-b782-be384795d961-istiod-ca-cert\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.707692 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.707597    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-workload-socket\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.707692 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.707661    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37aa2b03-ecca-44a1-b782-be384795d961-istiod-ca-cert\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.707899 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.707705    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-credential-socket\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.707899 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.707741    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-istio-envoy\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.707899 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.707784    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-istio-data\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.707899 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.707811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37aa2b03-ecca-44a1-b782-be384795d961-istio-podinfo\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.707899 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.707848    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-workload-certs\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.708157 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.707920    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65whr\" (UniqueName: \"kubernetes.io/projected/37aa2b03-ecca-44a1-b782-be384795d961-kube-api-access-65whr\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.708157 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.707953    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37aa2b03-ecca-44a1-b782-be384795d961-istio-token\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.708157 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.708095    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-workload-socket\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.708322 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.708196    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-istio-data\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.708555 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.708467    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-workload-certs\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.708555 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.708529    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37aa2b03-ecca-44a1-b782-be384795d961-istiod-ca-cert\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.708808 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.708644    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-credential-socket\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.710286 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.710266    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-istio-envoy\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.710675 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.710628    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37aa2b03-ecca-44a1-b782-be384795d961-istio-podinfo\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.717178 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.717156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37aa2b03-ecca-44a1-b782-be384795d961-istio-token\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.717382 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.717360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65whr\" (UniqueName: \"kubernetes.io/projected/37aa2b03-ecca-44a1-b782-be384795d961-kube-api-access-65whr\") pod \"e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") " pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.762296 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.762258    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:36.925010 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.924983    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"]
Jul 27 20:11:36.927725 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:11:36.927696    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37aa2b03_ecca_44a1_b782_be384795d961.slice/crio-dab4bde1ca40f6ff6d6bc573c0ca422b6f2d2db4c0bea98e2bf8c1d3ca1159a1 WatchSource:0}: Error finding container dab4bde1ca40f6ff6d6bc573c0ca422b6f2d2db4c0bea98e2bf8c1d3ca1159a1: Status 404 returned error can't find the container with id dab4bde1ca40f6ff6d6bc573c0ca422b6f2d2db4c0bea98e2bf8c1d3ca1159a1
Jul 27 20:11:36.930060 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.930025    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:11:36.930140 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.930102    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:11:36.930140 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:36.930130    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:11:37.727628 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:37.727594    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff" event={"ID":"37aa2b03-ecca-44a1-b782-be384795d961","Type":"ContainerStarted","Data":"d05e5aff9f5972a2b027bee0faa0a175df9a5b6dfad71ee845cbe163ae1e959d"}
Jul 27 20:11:37.728010 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:37.727685    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff" event={"ID":"37aa2b03-ecca-44a1-b782-be384795d961","Type":"ContainerStarted","Data":"dab4bde1ca40f6ff6d6bc573c0ca422b6f2d2db4c0bea98e2bf8c1d3ca1159a1"}
Jul 27 20:11:37.752027 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:37.751980    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff" podStartSLOduration=1.751965567 podStartE2EDuration="1.751965567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:11:37.749396978 +0000 UTC m=+1912.301313397" watchObservedRunningTime="2026-07-27 20:11:37.751965567 +0000 UTC m=+1912.303881985"
Jul 27 20:11:37.763164 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:37.763134    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:37.768384 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:37.768366    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:38.733362 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:38.733325    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:38.734395 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:38.734374    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:39.838733 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:39.838700    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"]
Jul 27 20:11:39.839184 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:39.839175    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerName="istio-proxy"
Jul 27 20:11:39.839243 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:39.839187    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65f5bde-0c0b-4774-8a4d-c897a04d530a" containerName="istio-proxy"
Jul 27 20:11:39.842748 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:39.842727    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:11:39.844618 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:39.844598    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-25d7f4d7-serving-cert\""
Jul 27 20:11:39.852661 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:39.852596    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"]
Jul 27 20:11:39.941914 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:39.941879    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3830bb28-27a5-4547-a4a7-308dd2f16571-maas-api-tls\") pod \"maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578\" (UID: \"3830bb28-27a5-4547-a4a7-308dd2f16571\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:11:39.942120 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:39.941951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3830bb28-27a5-4547-a4a7-308dd2f16571-openshift-service-ca\") pod \"maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578\" (UID: \"3830bb28-27a5-4547-a4a7-308dd2f16571\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:11:39.942120 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:39.942001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ldhl\" (UniqueName: \"kubernetes.io/projected/3830bb28-27a5-4547-a4a7-308dd2f16571-kube-api-access-2ldhl\") pod \"maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578\" (UID: \"3830bb28-27a5-4547-a4a7-308dd2f16571\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:11:40.042647 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:40.042609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ldhl\" (UniqueName: \"kubernetes.io/projected/3830bb28-27a5-4547-a4a7-308dd2f16571-kube-api-access-2ldhl\") pod \"maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578\" (UID: \"3830bb28-27a5-4547-a4a7-308dd2f16571\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:11:40.042813 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:40.042666    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3830bb28-27a5-4547-a4a7-308dd2f16571-maas-api-tls\") pod \"maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578\" (UID: \"3830bb28-27a5-4547-a4a7-308dd2f16571\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:11:40.042813 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:40.042712    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3830bb28-27a5-4547-a4a7-308dd2f16571-openshift-service-ca\") pod \"maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578\" (UID: \"3830bb28-27a5-4547-a4a7-308dd2f16571\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:11:40.042895 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:11:40.042818    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-serving-cert: secret "maas-api-e2e-mt-25d7f4d7-serving-cert" not found
Jul 27 20:11:40.042895 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:11:40.042884    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3830bb28-27a5-4547-a4a7-308dd2f16571-maas-api-tls podName:3830bb28-27a5-4547-a4a7-308dd2f16571 nodeName:}" failed. No retries permitted until 2026-07-27 20:11:40.542864657 +0000 UTC m=+1915.094781247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3830bb28-27a5-4547-a4a7-308dd2f16571-maas-api-tls") pod "maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578" (UID: "3830bb28-27a5-4547-a4a7-308dd2f16571") : secret "maas-api-e2e-mt-25d7f4d7-serving-cert" not found
Jul 27 20:11:40.043304 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:40.043287    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3830bb28-27a5-4547-a4a7-308dd2f16571-openshift-service-ca\") pod \"maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578\" (UID: \"3830bb28-27a5-4547-a4a7-308dd2f16571\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:11:40.056192 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:40.056166    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ldhl\" (UniqueName: \"kubernetes.io/projected/3830bb28-27a5-4547-a4a7-308dd2f16571-kube-api-access-2ldhl\") pod \"maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578\" (UID: \"3830bb28-27a5-4547-a4a7-308dd2f16571\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:11:40.547124 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:40.547090    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3830bb28-27a5-4547-a4a7-308dd2f16571-maas-api-tls\") pod \"maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578\" (UID: \"3830bb28-27a5-4547-a4a7-308dd2f16571\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:11:40.549697 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:40.549676    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3830bb28-27a5-4547-a4a7-308dd2f16571-maas-api-tls\") pod \"maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578\" (UID: \"3830bb28-27a5-4547-a4a7-308dd2f16571\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:11:40.757434 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:40.757399    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:11:40.898058 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:40.898030    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"]
Jul 27 20:11:40.900043 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:11:40.900013    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3830bb28_27a5_4547_a4a7_308dd2f16571.slice/crio-3509525f6eb6afd26124cb98d8ccfce1e0f71190e5a10e2280cb184e35e5979c WatchSource:0}: Error finding container 3509525f6eb6afd26124cb98d8ccfce1e0f71190e5a10e2280cb184e35e5979c: Status 404 returned error can't find the container with id 3509525f6eb6afd26124cb98d8ccfce1e0f71190e5a10e2280cb184e35e5979c
Jul 27 20:11:40.901796 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:40.901776    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:11:41.752402 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:41.752358    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578" event={"ID":"3830bb28-27a5-4547-a4a7-308dd2f16571","Type":"ContainerStarted","Data":"511b55246de87d937051518af1bff961f8d0270433217acdf1731597d735c3d9"}
Jul 27 20:11:41.752631 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:41.752427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578" event={"ID":"3830bb28-27a5-4547-a4a7-308dd2f16571","Type":"ContainerStarted","Data":"3509525f6eb6afd26124cb98d8ccfce1e0f71190e5a10e2280cb184e35e5979c"}
Jul 27 20:11:41.752631 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:41.752459    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:11:41.770705 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:41.770659    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578" podStartSLOduration=2.3314343920000002 podStartE2EDuration="2.770643802s" totalImagesPullingTime="439.20941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:11:41.767988357 +0000 UTC m=+1916.319904778" watchObservedRunningTime="2026-07-27 20:11:41.770643802 +0000 UTC m=+1916.322560219"
Jul 27 20:11:47.764194 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:47.764163    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:11:54.182135 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:54.181931    2562 patch_prober.go:28] interesting pod/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:36826->10.133.0.62:15021: read: connection reset by peer" start-of-body=
Jul 27 20:11:54.182135 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:54.182011    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff" podUID="37aa2b03-ecca-44a1-b782-be384795d961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:36826->10.133.0.62:15021: read: connection reset by peer"
Jul 27 20:11:54.835684 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:54.835651    2562 generic.go:356] "Generic (PLEG): container finished" podID="37aa2b03-ecca-44a1-b782-be384795d961" containerID="d05e5aff9f5972a2b027bee0faa0a175df9a5b6dfad71ee845cbe163ae1e959d" exitCode=0
Jul 27 20:11:54.835857 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:54.835724    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff" event={"ID":"37aa2b03-ecca-44a1-b782-be384795d961","Type":"ContainerDied","Data":"d05e5aff9f5972a2b027bee0faa0a175df9a5b6dfad71ee845cbe163ae1e959d"}
Jul 27 20:11:54.836163 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:54.836150    2562 scope.go:117] "RemoveContainer" containerID="d05e5aff9f5972a2b027bee0faa0a175df9a5b6dfad71ee845cbe163ae1e959d"
Jul 27 20:11:54.837253 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:54.837227    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:11:54.837318 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:54.837293    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:11:54.837358 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:54.837335    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:11:55.846118 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:55.846072    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff" event={"ID":"37aa2b03-ecca-44a1-b782-be384795d961","Type":"ContainerStarted","Data":"09017d5f2ebc31f51898f79f4787824d44137f02af1b7a719289827eda0656bd"}
Jul 27 20:11:56.763122 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:56.763086    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:11:56.763370 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:56.763347    2562 patch_prober.go:28] interesting pod/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 27 20:11:56.763457 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:56.763402    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff" podUID="37aa2b03-ecca-44a1-b782-be384795d961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 27 20:11:57.762743 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:57.762698    2562 patch_prober.go:28] interesting pod/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 27 20:11:57.763179 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:57.762762    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff" podUID="37aa2b03-ecca-44a1-b782-be384795d961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 27 20:11:58.763553 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:58.763514    2562 patch_prober.go:28] interesting pod/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 27 20:11:58.763934 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:58.763571    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff" podUID="37aa2b03-ecca-44a1-b782-be384795d961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 27 20:11:59.763614 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:59.763581    2562 patch_prober.go:28] interesting pod/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 27 20:11:59.764079 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:11:59.763637    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff" podUID="37aa2b03-ecca-44a1-b782-be384795d961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 27 20:12:00.766309 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:00.766276    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:12:00.766718 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:00.766539    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:12:00.767254 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:00.767238    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:12:54.901982 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:54.901943    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"]
Jul 27 20:12:54.902584 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:54.902259    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578" podUID="3830bb28-27a5-4547-a4a7-308dd2f16571" containerName="maas-api" containerID="cri-o://511b55246de87d937051518af1bff961f8d0270433217acdf1731597d735c3d9" gracePeriod=30
Jul 27 20:12:55.148806 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:55.148775    2562 generic.go:356] "Generic (PLEG): container finished" podID="3830bb28-27a5-4547-a4a7-308dd2f16571" containerID="511b55246de87d937051518af1bff961f8d0270433217acdf1731597d735c3d9" exitCode=0
Jul 27 20:12:55.148955 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:55.148842    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578" event={"ID":"3830bb28-27a5-4547-a4a7-308dd2f16571","Type":"ContainerDied","Data":"511b55246de87d937051518af1bff961f8d0270433217acdf1731597d735c3d9"}
Jul 27 20:12:55.170852 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:55.170822    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:12:55.286660 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:55.286616    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3830bb28-27a5-4547-a4a7-308dd2f16571-openshift-service-ca\") pod \"3830bb28-27a5-4547-a4a7-308dd2f16571\" (UID: \"3830bb28-27a5-4547-a4a7-308dd2f16571\") "
Jul 27 20:12:55.286851 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:55.286685    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ldhl\" (UniqueName: \"kubernetes.io/projected/3830bb28-27a5-4547-a4a7-308dd2f16571-kube-api-access-2ldhl\") pod \"3830bb28-27a5-4547-a4a7-308dd2f16571\" (UID: \"3830bb28-27a5-4547-a4a7-308dd2f16571\") "
Jul 27 20:12:55.286851 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:55.286734    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3830bb28-27a5-4547-a4a7-308dd2f16571-maas-api-tls\") pod \"3830bb28-27a5-4547-a4a7-308dd2f16571\" (UID: \"3830bb28-27a5-4547-a4a7-308dd2f16571\") "
Jul 27 20:12:55.287031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:55.287008    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3830bb28-27a5-4547-a4a7-308dd2f16571-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3830bb28-27a5-4547-a4a7-308dd2f16571" (UID: "3830bb28-27a5-4547-a4a7-308dd2f16571"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:12:55.288894 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:55.288860    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3830bb28-27a5-4547-a4a7-308dd2f16571-kube-api-access-2ldhl" (OuterVolumeSpecName: "kube-api-access-2ldhl") pod "3830bb28-27a5-4547-a4a7-308dd2f16571" (UID: "3830bb28-27a5-4547-a4a7-308dd2f16571"). InnerVolumeSpecName "kube-api-access-2ldhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:12:55.288987 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:55.288963    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3830bb28-27a5-4547-a4a7-308dd2f16571-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3830bb28-27a5-4547-a4a7-308dd2f16571" (UID: "3830bb28-27a5-4547-a4a7-308dd2f16571"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:12:55.388226 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:55.388197    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ldhl\" (UniqueName: \"kubernetes.io/projected/3830bb28-27a5-4547-a4a7-308dd2f16571-kube-api-access-2ldhl\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:12:55.388226 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:55.388226    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3830bb28-27a5-4547-a4a7-308dd2f16571-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:12:55.388440 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:55.388250    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3830bb28-27a5-4547-a4a7-308dd2f16571-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:12:56.155539 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:56.155507    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"
Jul 27 20:12:56.155944 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:56.155511    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578" event={"ID":"3830bb28-27a5-4547-a4a7-308dd2f16571","Type":"ContainerDied","Data":"3509525f6eb6afd26124cb98d8ccfce1e0f71190e5a10e2280cb184e35e5979c"}
Jul 27 20:12:56.155944 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:56.155645    2562 scope.go:117] "RemoveContainer" containerID="511b55246de87d937051518af1bff961f8d0270433217acdf1731597d735c3d9"
Jul 27 20:12:56.190856 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:56.190823    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"]
Jul 27 20:12:56.194030 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:56.194001    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25d7f4d7-777bf5b449-nl578"]
Jul 27 20:12:57.991778 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:12:57.991741    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3830bb28-27a5-4547-a4a7-308dd2f16571" path="/var/lib/kubelet/pods/3830bb28-27a5-4547-a4a7-308dd2f16571/volumes"
Jul 27 20:13:50.762348 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:50.762304    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"]
Jul 27 20:13:50.762882 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:50.762525    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff" podUID="37aa2b03-ecca-44a1-b782-be384795d961" containerName="istio-proxy" containerID="cri-o://09017d5f2ebc31f51898f79f4787824d44137f02af1b7a719289827eda0656bd" gracePeriod=30
Jul 27 20:13:56.016611 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.016588    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:13:56.058450 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.058362    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37aa2b03-ecca-44a1-b782-be384795d961-istio-podinfo\") pod \"37aa2b03-ecca-44a1-b782-be384795d961\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") "
Jul 27 20:13:56.058450 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.058433    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37aa2b03-ecca-44a1-b782-be384795d961-istio-token\") pod \"37aa2b03-ecca-44a1-b782-be384795d961\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") "
Jul 27 20:13:56.058708 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.058592    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-istio-envoy\") pod \"37aa2b03-ecca-44a1-b782-be384795d961\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") "
Jul 27 20:13:56.058708 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.058621    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-istio-data\") pod \"37aa2b03-ecca-44a1-b782-be384795d961\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") "
Jul 27 20:13:56.058708 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.058644    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65whr\" (UniqueName: \"kubernetes.io/projected/37aa2b03-ecca-44a1-b782-be384795d961-kube-api-access-65whr\") pod \"37aa2b03-ecca-44a1-b782-be384795d961\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") "
Jul 27 20:13:56.058708 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.058683    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-credential-socket\") pod \"37aa2b03-ecca-44a1-b782-be384795d961\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") "
Jul 27 20:13:56.058918 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.058715    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-workload-certs\") pod \"37aa2b03-ecca-44a1-b782-be384795d961\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") "
Jul 27 20:13:56.058918 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.058792    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-workload-socket\") pod \"37aa2b03-ecca-44a1-b782-be384795d961\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") "
Jul 27 20:13:56.058918 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.058853    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37aa2b03-ecca-44a1-b782-be384795d961-istiod-ca-cert\") pod \"37aa2b03-ecca-44a1-b782-be384795d961\" (UID: \"37aa2b03-ecca-44a1-b782-be384795d961\") "
Jul 27 20:13:56.059099 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.059064    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "37aa2b03-ecca-44a1-b782-be384795d961" (UID: "37aa2b03-ecca-44a1-b782-be384795d961"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:13:56.059243 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.059224    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-credential-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:13:56.059415 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.059363    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "37aa2b03-ecca-44a1-b782-be384795d961" (UID: "37aa2b03-ecca-44a1-b782-be384795d961"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:13:56.059548 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.059446    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "37aa2b03-ecca-44a1-b782-be384795d961" (UID: "37aa2b03-ecca-44a1-b782-be384795d961"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:13:56.059634 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.059608    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-istio-data" (OuterVolumeSpecName: "istio-data") pod "37aa2b03-ecca-44a1-b782-be384795d961" (UID: "37aa2b03-ecca-44a1-b782-be384795d961"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:13:56.059849 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.059829    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37aa2b03-ecca-44a1-b782-be384795d961-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "37aa2b03-ecca-44a1-b782-be384795d961" (UID: "37aa2b03-ecca-44a1-b782-be384795d961"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:13:56.061419 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.061386    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "37aa2b03-ecca-44a1-b782-be384795d961" (UID: "37aa2b03-ecca-44a1-b782-be384795d961"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:13:56.061586 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.061556    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37aa2b03-ecca-44a1-b782-be384795d961-istio-token" (OuterVolumeSpecName: "istio-token") pod "37aa2b03-ecca-44a1-b782-be384795d961" (UID: "37aa2b03-ecca-44a1-b782-be384795d961"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:13:56.061705 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.061556    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37aa2b03-ecca-44a1-b782-be384795d961-kube-api-access-65whr" (OuterVolumeSpecName: "kube-api-access-65whr") pod "37aa2b03-ecca-44a1-b782-be384795d961" (UID: "37aa2b03-ecca-44a1-b782-be384795d961"). InnerVolumeSpecName "kube-api-access-65whr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:13:56.061866 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.061850    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37aa2b03-ecca-44a1-b782-be384795d961-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "37aa2b03-ecca-44a1-b782-be384795d961" (UID: "37aa2b03-ecca-44a1-b782-be384795d961"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:13:56.160782 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.160747    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37aa2b03-ecca-44a1-b782-be384795d961-istio-podinfo\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:13:56.160782 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.160776    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37aa2b03-ecca-44a1-b782-be384795d961-istio-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:13:56.160782 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.160785    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-istio-envoy\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:13:56.161039 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.160794    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-istio-data\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:13:56.161039 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.160803    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65whr\" (UniqueName: \"kubernetes.io/projected/37aa2b03-ecca-44a1-b782-be384795d961-kube-api-access-65whr\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:13:56.161039 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.160812    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-workload-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:13:56.161039 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.160820    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aa2b03-ecca-44a1-b782-be384795d961-workload-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:13:56.161039 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.160828    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37aa2b03-ecca-44a1-b782-be384795d961-istiod-ca-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:13:56.479085 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.479051    2562 generic.go:356] "Generic (PLEG): container finished" podID="37aa2b03-ecca-44a1-b782-be384795d961" containerID="09017d5f2ebc31f51898f79f4787824d44137f02af1b7a719289827eda0656bd" exitCode=0
Jul 27 20:13:56.479259 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.479096    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"
Jul 27 20:13:56.479259 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.479139    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff" event={"ID":"37aa2b03-ecca-44a1-b782-be384795d961","Type":"ContainerDied","Data":"09017d5f2ebc31f51898f79f4787824d44137f02af1b7a719289827eda0656bd"}
Jul 27 20:13:56.479259 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.479192    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff" event={"ID":"37aa2b03-ecca-44a1-b782-be384795d961","Type":"ContainerDied","Data":"dab4bde1ca40f6ff6d6bc573c0ca422b6f2d2db4c0bea98e2bf8c1d3ca1159a1"}
Jul 27 20:13:56.479259 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.479214    2562 scope.go:117] "RemoveContainer" containerID="09017d5f2ebc31f51898f79f4787824d44137f02af1b7a719289827eda0656bd"
Jul 27 20:13:56.488798 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.488780    2562 scope.go:117] "RemoveContainer" containerID="d05e5aff9f5972a2b027bee0faa0a175df9a5b6dfad71ee845cbe163ae1e959d"
Jul 27 20:13:56.500350 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.500315    2562 scope.go:117] "RemoveContainer" containerID="09017d5f2ebc31f51898f79f4787824d44137f02af1b7a719289827eda0656bd"
Jul 27 20:13:56.502650 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:13:56.502625    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09017d5f2ebc31f51898f79f4787824d44137f02af1b7a719289827eda0656bd\": container with ID starting with 09017d5f2ebc31f51898f79f4787824d44137f02af1b7a719289827eda0656bd not found: ID does not exist" containerID="09017d5f2ebc31f51898f79f4787824d44137f02af1b7a719289827eda0656bd"
Jul 27 20:13:56.502774 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.502661    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09017d5f2ebc31f51898f79f4787824d44137f02af1b7a719289827eda0656bd"} err="failed to get container status \"09017d5f2ebc31f51898f79f4787824d44137f02af1b7a719289827eda0656bd\": rpc error: code = NotFound desc = could not find container \"09017d5f2ebc31f51898f79f4787824d44137f02af1b7a719289827eda0656bd\": container with ID starting with 09017d5f2ebc31f51898f79f4787824d44137f02af1b7a719289827eda0656bd not found: ID does not exist"
Jul 27 20:13:56.502774 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.502685    2562 scope.go:117] "RemoveContainer" containerID="d05e5aff9f5972a2b027bee0faa0a175df9a5b6dfad71ee845cbe163ae1e959d"
Jul 27 20:13:56.502890 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.502811    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"]
Jul 27 20:13:56.502960 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:13:56.502938    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05e5aff9f5972a2b027bee0faa0a175df9a5b6dfad71ee845cbe163ae1e959d\": container with ID starting with d05e5aff9f5972a2b027bee0faa0a175df9a5b6dfad71ee845cbe163ae1e959d not found: ID does not exist" containerID="d05e5aff9f5972a2b027bee0faa0a175df9a5b6dfad71ee845cbe163ae1e959d"
Jul 27 20:13:56.503010 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.502969    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05e5aff9f5972a2b027bee0faa0a175df9a5b6dfad71ee845cbe163ae1e959d"} err="failed to get container status \"d05e5aff9f5972a2b027bee0faa0a175df9a5b6dfad71ee845cbe163ae1e959d\": rpc error: code = NotFound desc = could not find container \"d05e5aff9f5972a2b027bee0faa0a175df9a5b6dfad71ee845cbe163ae1e959d\": container with ID starting with d05e5aff9f5972a2b027bee0faa0a175df9a5b6dfad71ee845cbe163ae1e959d not found: ID does not exist"
Jul 27 20:13:56.507566 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:56.507546    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-25d7f4d7-openshift-default-86bd4b6564-dzcff"]
Jul 27 20:13:57.991463 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:13:57.991428    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37aa2b03-ecca-44a1-b782-be384795d961" path="/var/lib/kubelet/pods/37aa2b03-ecca-44a1-b782-be384795d961/volumes"
Jul 27 20:14:46.988473 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:46.988426    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"]
Jul 27 20:14:46.988928 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:46.988913    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37aa2b03-ecca-44a1-b782-be384795d961" containerName="istio-proxy"
Jul 27 20:14:46.988972 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:46.988930    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="37aa2b03-ecca-44a1-b782-be384795d961" containerName="istio-proxy"
Jul 27 20:14:46.988972 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:46.988954    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3830bb28-27a5-4547-a4a7-308dd2f16571" containerName="maas-api"
Jul 27 20:14:46.988972 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:46.988960    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3830bb28-27a5-4547-a4a7-308dd2f16571" containerName="maas-api"
Jul 27 20:14:46.989068 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:46.989015    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="37aa2b03-ecca-44a1-b782-be384795d961" containerName="istio-proxy"
Jul 27 20:14:46.989068 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:46.989022    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3830bb28-27a5-4547-a4a7-308dd2f16571" containerName="maas-api"
Jul 27 20:14:46.993714 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:46.993692    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:46.996545 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:46.996527    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-849f6b-openshift-default-dockercfg-5pgsh\""
Jul 27 20:14:47.010686 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.010644    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"]
Jul 27 20:14:47.046757 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.046727    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-credential-socket\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.046912 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.046780    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-workload-socket\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.046912 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.046799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9cr\" (UniqueName: \"kubernetes.io/projected/1465e850-eb74-4e35-9548-6564cb3ff299-kube-api-access-qp9cr\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.046912 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.046865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-istio-data\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.046912 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.046892    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1465e850-eb74-4e35-9548-6564cb3ff299-istio-token\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.047054 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.046971    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-workload-certs\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.047054 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.047003    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1465e850-eb74-4e35-9548-6564cb3ff299-istiod-ca-cert\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.047054 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.047034    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-istio-envoy\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.047054 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.047052    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1465e850-eb74-4e35-9548-6564cb3ff299-istio-podinfo\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.147544 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.147502    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-workload-certs\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.147544 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.147551    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1465e850-eb74-4e35-9548-6564cb3ff299-istiod-ca-cert\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.147812 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.147582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-istio-envoy\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.147812 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.147600    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1465e850-eb74-4e35-9548-6564cb3ff299-istio-podinfo\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.147812 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.147622    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-credential-socket\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.147812 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.147660    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-workload-socket\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.148019 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.147971    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-workload-certs\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.148131 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.148106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9cr\" (UniqueName: \"kubernetes.io/projected/1465e850-eb74-4e35-9548-6564cb3ff299-kube-api-access-qp9cr\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.148192 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.148140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-credential-socket\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.148192 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.148165    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-istio-data\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.148302 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.148200    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1465e850-eb74-4e35-9548-6564cb3ff299-istio-token\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.148355 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.148323    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1465e850-eb74-4e35-9548-6564cb3ff299-istiod-ca-cert\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.148513 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.148490    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-istio-data\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.148604 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.148585    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-workload-socket\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.150459 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.150442    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-istio-envoy\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.150786 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.150764    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1465e850-eb74-4e35-9548-6564cb3ff299-istio-podinfo\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.157327 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.157298    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1465e850-eb74-4e35-9548-6564cb3ff299-istio-token\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.157428 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.157401    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9cr\" (UniqueName: \"kubernetes.io/projected/1465e850-eb74-4e35-9548-6564cb3ff299-kube-api-access-qp9cr\") pod \"e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") " pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.306344 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.306255    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:47.466066 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.466035    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"]
Jul 27 20:14:47.468899 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:14:47.468871    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1465e850_eb74_4e35_9548_6564cb3ff299.slice/crio-757f99eb77392a79221c7e935316f305fd5b72081a8d4757b6adbdc7034c440d WatchSource:0}: Error finding container 757f99eb77392a79221c7e935316f305fd5b72081a8d4757b6adbdc7034c440d: Status 404 returned error can't find the container with id 757f99eb77392a79221c7e935316f305fd5b72081a8d4757b6adbdc7034c440d
Jul 27 20:14:47.471069 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.471037    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:14:47.471141 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.471109    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:14:47.471196 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.471139    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:14:47.746839 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.746796    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm" event={"ID":"1465e850-eb74-4e35-9548-6564cb3ff299","Type":"ContainerStarted","Data":"266b84abf592301afffee3349991c839221b8ca539353a5e8b018d57956daf34"}
Jul 27 20:14:47.747044 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.746867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm" event={"ID":"1465e850-eb74-4e35-9548-6564cb3ff299","Type":"ContainerStarted","Data":"757f99eb77392a79221c7e935316f305fd5b72081a8d4757b6adbdc7034c440d"}
Jul 27 20:14:47.766422 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:47.766333    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm" podStartSLOduration=1.7663125160000002 podStartE2EDuration="1.766312516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:47.765733514 +0000 UTC m=+2102.317649934" watchObservedRunningTime="2026-07-27 20:14:47.766312516 +0000 UTC m=+2102.318228935"
Jul 27 20:14:48.307015 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:48.306976    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:48.311789 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:48.311761    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:48.751634 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:48.751605    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:48.752545 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:48.752527    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:14:49.797130 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.797049    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"]
Jul 27 20:14:49.797714 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.797697    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37aa2b03-ecca-44a1-b782-be384795d961" containerName="istio-proxy"
Jul 27 20:14:49.797800 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.797716    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="37aa2b03-ecca-44a1-b782-be384795d961" containerName="istio-proxy"
Jul 27 20:14:49.797800 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.797794    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="37aa2b03-ecca-44a1-b782-be384795d961" containerName="istio-proxy"
Jul 27 20:14:49.800815 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.800797    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:14:49.803044 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.803022    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-849f6b-serving-cert\""
Jul 27 20:14:49.812216 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.812188    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"]
Jul 27 20:14:49.878504 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.878453    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs8wj\" (UniqueName: \"kubernetes.io/projected/bb3dcd21-d7e7-49cf-8b65-010ada3208df-kube-api-access-rs8wj\") pod \"maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf\" (UID: \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:14:49.878695 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.878552    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3dcd21-d7e7-49cf-8b65-010ada3208df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf\" (UID: \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:14:49.878695 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.878666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3dcd21-d7e7-49cf-8b65-010ada3208df-maas-api-tls\") pod \"maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf\" (UID: \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:14:49.979653 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.979620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs8wj\" (UniqueName: \"kubernetes.io/projected/bb3dcd21-d7e7-49cf-8b65-010ada3208df-kube-api-access-rs8wj\") pod \"maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf\" (UID: \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:14:49.979857 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.979667    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3dcd21-d7e7-49cf-8b65-010ada3208df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf\" (UID: \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:14:49.979857 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.979733    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3dcd21-d7e7-49cf-8b65-010ada3208df-maas-api-tls\") pod \"maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf\" (UID: \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:14:49.979857 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:14:49.979841    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-serving-cert: secret "maas-api-e2e-inf-a-849f6b-serving-cert" not found
Jul 27 20:14:49.980046 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:14:49.979936    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3dcd21-d7e7-49cf-8b65-010ada3208df-maas-api-tls podName:bb3dcd21-d7e7-49cf-8b65-010ada3208df nodeName:}" failed. No retries permitted until 2026-07-27 20:14:50.479913233 +0000 UTC m=+2105.031829634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bb3dcd21-d7e7-49cf-8b65-010ada3208df-maas-api-tls") pod "maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf" (UID: "bb3dcd21-d7e7-49cf-8b65-010ada3208df") : secret "maas-api-e2e-inf-a-849f6b-serving-cert" not found
Jul 27 20:14:49.980446 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.980419    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3dcd21-d7e7-49cf-8b65-010ada3208df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf\" (UID: \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:14:49.991656 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:49.991631    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs8wj\" (UniqueName: \"kubernetes.io/projected/bb3dcd21-d7e7-49cf-8b65-010ada3208df-kube-api-access-rs8wj\") pod \"maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf\" (UID: \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:14:50.484565 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:50.484523    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3dcd21-d7e7-49cf-8b65-010ada3208df-maas-api-tls\") pod \"maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf\" (UID: \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:14:50.487079 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:50.487058    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3dcd21-d7e7-49cf-8b65-010ada3208df-maas-api-tls\") pod \"maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf\" (UID: \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:14:50.715977 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:50.715942    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:14:50.862067 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:50.862041    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"]
Jul 27 20:14:50.863695 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:14:50.863670    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3dcd21_d7e7_49cf_8b65_010ada3208df.slice/crio-00453a92db663a98ccec65d1422450a49282245af67a138f3a0e40d9d8315e38 WatchSource:0}: Error finding container 00453a92db663a98ccec65d1422450a49282245af67a138f3a0e40d9d8315e38: Status 404 returned error can't find the container with id 00453a92db663a98ccec65d1422450a49282245af67a138f3a0e40d9d8315e38
Jul 27 20:14:51.768648 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:51.768609    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf" event={"ID":"bb3dcd21-d7e7-49cf-8b65-010ada3208df","Type":"ContainerStarted","Data":"abffc376d10075fd525ec7a220fd665ccfc2a35b4dbb1cee2fcc46dc1247b7ad"}
Jul 27 20:14:51.768935 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:51.768688    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf" event={"ID":"bb3dcd21-d7e7-49cf-8b65-010ada3208df","Type":"ContainerStarted","Data":"00453a92db663a98ccec65d1422450a49282245af67a138f3a0e40d9d8315e38"}
Jul 27 20:14:51.768935 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:51.768765    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:14:51.801697 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:51.801640    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf" podStartSLOduration=2.329341738 podStartE2EDuration="2.80162575s" totalImagesPullingTime="472.284012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:51.798196046 +0000 UTC m=+2106.350112475" watchObservedRunningTime="2026-07-27 20:14:51.80162575 +0000 UTC m=+2106.353542190"
Jul 27 20:14:54.797777 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:54.797738    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"]
Jul 27 20:14:54.802336 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:54.802313    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:54.804636 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:54.804615    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0a5392-openshift-default-dockercfg-m9b8b\""
Jul 27 20:14:54.814148 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:54.814125    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"]
Jul 27 20:14:54.933969 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:54.933933    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9262cb-5152-460c-bdbc-d65acb551c43-istiod-ca-cert\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:54.934172 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:54.933982    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lpgt\" (UniqueName: \"kubernetes.io/projected/3a9262cb-5152-460c-bdbc-d65acb551c43-kube-api-access-2lpgt\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:54.934172 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:54.934016    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-credential-socket\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:54.934172 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:54.934032    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-envoy\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:54.934172 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:54.934080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-podinfo\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:54.934172 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:54.934109    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-token\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:54.934382 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:54.934189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-workload-certs\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:54.934382 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:54.934210    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-data\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:54.934382 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:54.934235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-workload-socket\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.035727 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.035686    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-podinfo\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.035928 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.035750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-token\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.035928 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.035806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-workload-certs\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.035928 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.035828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-data\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.035928 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.035858    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-workload-socket\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.036152 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.035927    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9262cb-5152-460c-bdbc-d65acb551c43-istiod-ca-cert\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.036152 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.035987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lpgt\" (UniqueName: \"kubernetes.io/projected/3a9262cb-5152-460c-bdbc-d65acb551c43-kube-api-access-2lpgt\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.036152 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.036027    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-credential-socket\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.036152 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.036056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-envoy\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.036390 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.036363    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-workload-socket\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.036547 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.036407    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-data\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.036547 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.036473    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-credential-socket\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.036691 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.036669    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-workload-certs\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.036781 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.036763    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9262cb-5152-460c-bdbc-d65acb551c43-istiod-ca-cert\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.038747 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.038728    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-podinfo\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.038815 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.038764    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-envoy\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.044644 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.044614    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lpgt\" (UniqueName: \"kubernetes.io/projected/3a9262cb-5152-460c-bdbc-d65acb551c43-kube-api-access-2lpgt\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.044743 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.044617    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-token\") pod \"e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") " pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.113905 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.113813    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:55.264854 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.264828    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"]
Jul 27 20:14:55.266515 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:14:55.266460    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9262cb_5152_460c_bdbc_d65acb551c43.slice/crio-a6b664dd37b427583993293f712377613460256d8ae190030e018fbbca5c84e9 WatchSource:0}: Error finding container a6b664dd37b427583993293f712377613460256d8ae190030e018fbbca5c84e9: Status 404 returned error can't find the container with id a6b664dd37b427583993293f712377613460256d8ae190030e018fbbca5c84e9
Jul 27 20:14:55.268824 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.268792    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:14:55.268899 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.268863    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:14:55.268899 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.268893    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:14:55.794985 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.794952    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr" event={"ID":"3a9262cb-5152-460c-bdbc-d65acb551c43","Type":"ContainerStarted","Data":"5380a265c59111e3da2fddc9caec35775e32b8a9122adbc7dd82d9cca7b68d84"}
Jul 27 20:14:55.795178 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.795002    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr" event={"ID":"3a9262cb-5152-460c-bdbc-d65acb551c43","Type":"ContainerStarted","Data":"a6b664dd37b427583993293f712377613460256d8ae190030e018fbbca5c84e9"}
Jul 27 20:14:55.819192 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:55.819136    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr" podStartSLOduration=1.819118099 podStartE2EDuration="1.819118099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:55.817314252 +0000 UTC m=+2110.369230683" watchObservedRunningTime="2026-07-27 20:14:55.819118099 +0000 UTC m=+2110.371034575"
Jul 27 20:14:56.114541 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:56.114424    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:56.119249 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:56.119224    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:56.806754 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:56.806709    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:56.807620 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:56.807597    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:14:57.780004 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:57.779975    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:14:57.851174 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:57.851128    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"]
Jul 27 20:14:57.855921 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:57.855895    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:14:57.857909 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:57.857888    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0a5392-serving-cert\""
Jul 27 20:14:57.865338 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:57.865305    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"]
Jul 27 20:14:57.967296 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:57.967261    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl\" (UID: \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:14:57.967640 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:57.967603    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl\" (UID: \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:14:57.967861 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:57.967841    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zr85\" (UniqueName: \"kubernetes.io/projected/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-kube-api-access-7zr85\") pod \"maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl\" (UID: \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:14:58.069435 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:58.069345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl\" (UID: \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:14:58.069435 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:58.069390    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl\" (UID: \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:14:58.069709 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:14:58.069522    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-serving-cert: secret "maas-api-e2e-inf-b-0a5392-serving-cert" not found
Jul 27 20:14:58.069709 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:58.069527    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zr85\" (UniqueName: \"kubernetes.io/projected/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-kube-api-access-7zr85\") pod \"maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl\" (UID: \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:14:58.069709 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:14:58.069585    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-maas-api-tls podName:ce195c9f-1aad-4439-ba44-e6dffa27a6f7 nodeName:}" failed. No retries permitted until 2026-07-27 20:14:58.5695686 +0000 UTC m=+2113.121484996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-maas-api-tls") pod "maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl" (UID: "ce195c9f-1aad-4439-ba44-e6dffa27a6f7") : secret "maas-api-e2e-inf-b-0a5392-serving-cert" not found
Jul 27 20:14:58.070044 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:58.070027    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl\" (UID: \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:14:58.080986 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:58.080948    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zr85\" (UniqueName: \"kubernetes.io/projected/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-kube-api-access-7zr85\") pod \"maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl\" (UID: \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:14:58.594507 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:58.589311    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl\" (UID: \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:14:58.606133 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:58.606066    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl\" (UID: \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:14:58.770759 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:58.770644    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:14:58.968719 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:58.968555    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"]
Jul 27 20:14:58.970977 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:14:58.970935    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce195c9f_1aad_4439_ba44_e6dffa27a6f7.slice/crio-c93e5ff15378c77ed0207175026c4a3fad430ccf57142bd9239f18af24ca1caa WatchSource:0}: Error finding container c93e5ff15378c77ed0207175026c4a3fad430ccf57142bd9239f18af24ca1caa: Status 404 returned error can't find the container with id c93e5ff15378c77ed0207175026c4a3fad430ccf57142bd9239f18af24ca1caa
Jul 27 20:14:59.827445 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:59.827407    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl" event={"ID":"ce195c9f-1aad-4439-ba44-e6dffa27a6f7","Type":"ContainerStarted","Data":"e43fdcb680602f3a09cbb896f3a16007710c417f8c4ed8d6575c14ecf07759a5"}
Jul 27 20:14:59.827645 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:59.827460    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl" event={"ID":"ce195c9f-1aad-4439-ba44-e6dffa27a6f7","Type":"ContainerStarted","Data":"c93e5ff15378c77ed0207175026c4a3fad430ccf57142bd9239f18af24ca1caa"}
Jul 27 20:14:59.827645 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:59.827537    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:14:59.846749 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:14:59.846692    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl" podStartSLOduration=2.476277445 podStartE2EDuration="2.846674815s" totalImagesPullingTime="370.39737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:59.844385276 +0000 UTC m=+2114.396301694" watchObservedRunningTime="2026-07-27 20:14:59.846674815 +0000 UTC m=+2114.398591233"
Jul 27 20:15:04.793686 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:04.793643    2562 patch_prober.go:28] interesting pod/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 20:15:04.794108 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:04.793717    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm" podUID="1465e850-eb74-4e35-9548-6564cb3ff299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 20:15:05.837998 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:05.837898    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:15:05.862263 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:05.862231    2562 generic.go:356] "Generic (PLEG): container finished" podID="1465e850-eb74-4e35-9548-6564cb3ff299" containerID="266b84abf592301afffee3349991c839221b8ca539353a5e8b018d57956daf34" exitCode=0
Jul 27 20:15:05.862469 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:05.862278    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm" event={"ID":"1465e850-eb74-4e35-9548-6564cb3ff299","Type":"ContainerDied","Data":"266b84abf592301afffee3349991c839221b8ca539353a5e8b018d57956daf34"}
Jul 27 20:15:05.862719 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:05.862701    2562 scope.go:117] "RemoveContainer" containerID="266b84abf592301afffee3349991c839221b8ca539353a5e8b018d57956daf34"
Jul 27 20:15:05.863915 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:05.863883    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:15:05.863996 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:05.863956    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:15:05.864054 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:05.863994    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:15:06.895113 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:06.895073    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm" event={"ID":"1465e850-eb74-4e35-9548-6564cb3ff299","Type":"ContainerStarted","Data":"5f837e7dfe903fe1fc37230e738a675753bcacb2e9875bd927afec2b5b79ac52"}
Jul 27 20:15:07.306900 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:07.306697    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:15:07.307263 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:07.307236    2562 patch_prober.go:28] interesting pod/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 27 20:15:07.307471 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:07.307440    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm" podUID="1465e850-eb74-4e35-9548-6564cb3ff299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 27 20:15:08.307783 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:08.307736    2562 patch_prober.go:28] interesting pod/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 27 20:15:08.308301 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:08.307839    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm" podUID="1465e850-eb74-4e35-9548-6564cb3ff299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 27 20:15:09.307322 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:09.307277    2562 patch_prober.go:28] interesting pod/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 27 20:15:09.307530 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:09.307347    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm" podUID="1465e850-eb74-4e35-9548-6564cb3ff299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 27 20:15:10.307627 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:10.307588    2562 patch_prober.go:28] interesting pod/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 27 20:15:10.308218 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:10.308182    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm" podUID="1465e850-eb74-4e35-9548-6564cb3ff299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 27 20:15:11.310625 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:11.310534    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:15:11.311312 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:11.311284    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:15:11.312044 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:11.312028    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:15:12.848678 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:12.848635    2562 patch_prober.go:28] interesting pod/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 20:15:12.849137 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:12.848699    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr" podUID="3a9262cb-5152-460c-bdbc-d65acb551c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 20:15:13.946893 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:13.946859    2562 generic.go:356] "Generic (PLEG): container finished" podID="3a9262cb-5152-460c-bdbc-d65acb551c43" containerID="5380a265c59111e3da2fddc9caec35775e32b8a9122adbc7dd82d9cca7b68d84" exitCode=0
Jul 27 20:15:13.947309 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:13.946912    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr" event={"ID":"3a9262cb-5152-460c-bdbc-d65acb551c43","Type":"ContainerDied","Data":"5380a265c59111e3da2fddc9caec35775e32b8a9122adbc7dd82d9cca7b68d84"}
Jul 27 20:15:13.947352 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:13.947331    2562 scope.go:117] "RemoveContainer" containerID="5380a265c59111e3da2fddc9caec35775e32b8a9122adbc7dd82d9cca7b68d84"
Jul 27 20:15:13.948468 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:13.948427    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:15:13.948575 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:13.948526    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:15:13.948575 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:13.948561    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:15:14.954833 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:14.954795    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr" event={"ID":"3a9262cb-5152-460c-bdbc-d65acb551c43","Type":"ContainerStarted","Data":"cf633e48e27220cc190c51d3dda271e64f2f9413c74a50105b135283007c5730"}
Jul 27 20:15:15.114502 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:15.114435    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:15:15.114736 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:15.114713    2562 patch_prober.go:28] interesting pod/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 27 20:15:15.114831 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:15.114805    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr" podUID="3a9262cb-5152-460c-bdbc-d65acb551c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 27 20:15:16.114246 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:16.114211    2562 patch_prober.go:28] interesting pod/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 27 20:15:16.114649 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:16.114260    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr" podUID="3a9262cb-5152-460c-bdbc-d65acb551c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 27 20:15:17.114546 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:17.114508    2562 patch_prober.go:28] interesting pod/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 27 20:15:17.115054 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:17.114574    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr" podUID="3a9262cb-5152-460c-bdbc-d65acb551c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 27 20:15:18.115046 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:18.115002    2562 patch_prober.go:28] interesting pod/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 27 20:15:18.115437 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:18.115061    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr" podUID="3a9262cb-5152-460c-bdbc-d65acb551c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 27 20:15:19.118398 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:19.118364    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:15:19.118797 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:19.118748    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:15:19.119360 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:15:19.119345    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:16:22.998684 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:22.998651    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"]
Jul 27 20:16:22.999295 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:22.998874    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf" podUID="bb3dcd21-d7e7-49cf-8b65-010ada3208df" containerName="maas-api" containerID="cri-o://abffc376d10075fd525ec7a220fd665ccfc2a35b4dbb1cee2fcc46dc1247b7ad" gracePeriod=30
Jul 27 20:16:23.271206 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.271177    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:16:23.330184 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.330151    2562 generic.go:356] "Generic (PLEG): container finished" podID="bb3dcd21-d7e7-49cf-8b65-010ada3208df" containerID="abffc376d10075fd525ec7a220fd665ccfc2a35b4dbb1cee2fcc46dc1247b7ad" exitCode=0
Jul 27 20:16:23.330354 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.330209    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"
Jul 27 20:16:23.330354 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.330231    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf" event={"ID":"bb3dcd21-d7e7-49cf-8b65-010ada3208df","Type":"ContainerDied","Data":"abffc376d10075fd525ec7a220fd665ccfc2a35b4dbb1cee2fcc46dc1247b7ad"}
Jul 27 20:16:23.330354 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.330275    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf" event={"ID":"bb3dcd21-d7e7-49cf-8b65-010ada3208df","Type":"ContainerDied","Data":"00453a92db663a98ccec65d1422450a49282245af67a138f3a0e40d9d8315e38"}
Jul 27 20:16:23.330354 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.330296    2562 scope.go:117] "RemoveContainer" containerID="abffc376d10075fd525ec7a220fd665ccfc2a35b4dbb1cee2fcc46dc1247b7ad"
Jul 27 20:16:23.339347 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.339331    2562 scope.go:117] "RemoveContainer" containerID="abffc376d10075fd525ec7a220fd665ccfc2a35b4dbb1cee2fcc46dc1247b7ad"
Jul 27 20:16:23.339659 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:16:23.339638    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abffc376d10075fd525ec7a220fd665ccfc2a35b4dbb1cee2fcc46dc1247b7ad\": container with ID starting with abffc376d10075fd525ec7a220fd665ccfc2a35b4dbb1cee2fcc46dc1247b7ad not found: ID does not exist" containerID="abffc376d10075fd525ec7a220fd665ccfc2a35b4dbb1cee2fcc46dc1247b7ad"
Jul 27 20:16:23.339771 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.339668    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abffc376d10075fd525ec7a220fd665ccfc2a35b4dbb1cee2fcc46dc1247b7ad"} err="failed to get container status \"abffc376d10075fd525ec7a220fd665ccfc2a35b4dbb1cee2fcc46dc1247b7ad\": rpc error: code = NotFound desc = could not find container \"abffc376d10075fd525ec7a220fd665ccfc2a35b4dbb1cee2fcc46dc1247b7ad\": container with ID starting with abffc376d10075fd525ec7a220fd665ccfc2a35b4dbb1cee2fcc46dc1247b7ad not found: ID does not exist"
Jul 27 20:16:23.404933 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.404903    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3dcd21-d7e7-49cf-8b65-010ada3208df-maas-api-tls\") pod \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\" (UID: \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\") "
Jul 27 20:16:23.405105 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.404972    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3dcd21-d7e7-49cf-8b65-010ada3208df-openshift-service-ca\") pod \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\" (UID: \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\") "
Jul 27 20:16:23.405105 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.404997    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs8wj\" (UniqueName: \"kubernetes.io/projected/bb3dcd21-d7e7-49cf-8b65-010ada3208df-kube-api-access-rs8wj\") pod \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\" (UID: \"bb3dcd21-d7e7-49cf-8b65-010ada3208df\") "
Jul 27 20:16:23.405355 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.405331    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3dcd21-d7e7-49cf-8b65-010ada3208df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb3dcd21-d7e7-49cf-8b65-010ada3208df" (UID: "bb3dcd21-d7e7-49cf-8b65-010ada3208df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:16:23.407195 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.407169    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3dcd21-d7e7-49cf-8b65-010ada3208df-kube-api-access-rs8wj" (OuterVolumeSpecName: "kube-api-access-rs8wj") pod "bb3dcd21-d7e7-49cf-8b65-010ada3208df" (UID: "bb3dcd21-d7e7-49cf-8b65-010ada3208df"). InnerVolumeSpecName "kube-api-access-rs8wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:16:23.407381 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.407362    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3dcd21-d7e7-49cf-8b65-010ada3208df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb3dcd21-d7e7-49cf-8b65-010ada3208df" (UID: "bb3dcd21-d7e7-49cf-8b65-010ada3208df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:16:23.506902 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.506823    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3dcd21-d7e7-49cf-8b65-010ada3208df-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:23.506902 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.506853    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3dcd21-d7e7-49cf-8b65-010ada3208df-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:23.506902 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.506864    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs8wj\" (UniqueName: \"kubernetes.io/projected/bb3dcd21-d7e7-49cf-8b65-010ada3208df-kube-api-access-rs8wj\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:23.651138 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.651109    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"]
Jul 27 20:16:23.655014 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.654991    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-849f6b-65d966b6bf-z4ppf"]
Jul 27 20:16:23.991741 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:23.991706    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb3dcd21-d7e7-49cf-8b65-010ada3208df" path="/var/lib/kubelet/pods/bb3dcd21-d7e7-49cf-8b65-010ada3208df/volumes"
Jul 27 20:16:34.104418 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:34.104384    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"]
Jul 27 20:16:34.104906 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:34.104659    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm" podUID="1465e850-eb74-4e35-9548-6564cb3ff299" containerName="istio-proxy" containerID="cri-o://5f837e7dfe903fe1fc37230e738a675753bcacb2e9875bd927afec2b5b79ac52" gracePeriod=30
Jul 27 20:16:39.361561 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.361534    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:16:39.363259 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363239    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-credential-socket\") pod \"1465e850-eb74-4e35-9548-6564cb3ff299\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") "
Jul 27 20:16:39.363372 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363295    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1465e850-eb74-4e35-9548-6564cb3ff299-istio-podinfo\") pod \"1465e850-eb74-4e35-9548-6564cb3ff299\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") "
Jul 27 20:16:39.363372 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363328    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1465e850-eb74-4e35-9548-6564cb3ff299-istio-token\") pod \"1465e850-eb74-4e35-9548-6564cb3ff299\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") "
Jul 27 20:16:39.363505 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363375    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-workload-certs\") pod \"1465e850-eb74-4e35-9548-6564cb3ff299\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") "
Jul 27 20:16:39.363505 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363400    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-workload-socket\") pod \"1465e850-eb74-4e35-9548-6564cb3ff299\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") "
Jul 27 20:16:39.363505 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363433    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-istio-data\") pod \"1465e850-eb74-4e35-9548-6564cb3ff299\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") "
Jul 27 20:16:39.363505 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363464    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp9cr\" (UniqueName: \"kubernetes.io/projected/1465e850-eb74-4e35-9548-6564cb3ff299-kube-api-access-qp9cr\") pod \"1465e850-eb74-4e35-9548-6564cb3ff299\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") "
Jul 27 20:16:39.363716 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363534    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-istio-envoy\") pod \"1465e850-eb74-4e35-9548-6564cb3ff299\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") "
Jul 27 20:16:39.363716 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363533    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1465e850-eb74-4e35-9548-6564cb3ff299" (UID: "1465e850-eb74-4e35-9548-6564cb3ff299"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:16:39.363716 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363570    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1465e850-eb74-4e35-9548-6564cb3ff299-istiod-ca-cert\") pod \"1465e850-eb74-4e35-9548-6564cb3ff299\" (UID: \"1465e850-eb74-4e35-9548-6564cb3ff299\") "
Jul 27 20:16:39.363860 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363710    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1465e850-eb74-4e35-9548-6564cb3ff299" (UID: "1465e850-eb74-4e35-9548-6564cb3ff299"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:16:39.363860 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363720    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1465e850-eb74-4e35-9548-6564cb3ff299" (UID: "1465e850-eb74-4e35-9548-6564cb3ff299"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:16:39.363968 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363939    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-workload-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:39.363968 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363957    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-workload-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:39.364054 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.363971    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-credential-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:39.364054 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.364012    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1465e850-eb74-4e35-9548-6564cb3ff299-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1465e850-eb74-4e35-9548-6564cb3ff299" (UID: "1465e850-eb74-4e35-9548-6564cb3ff299"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:16:39.364566 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.364539    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-istio-data" (OuterVolumeSpecName: "istio-data") pod "1465e850-eb74-4e35-9548-6564cb3ff299" (UID: "1465e850-eb74-4e35-9548-6564cb3ff299"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:16:39.365942 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.365877    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1465e850-eb74-4e35-9548-6564cb3ff299-istio-token" (OuterVolumeSpecName: "istio-token") pod "1465e850-eb74-4e35-9548-6564cb3ff299" (UID: "1465e850-eb74-4e35-9548-6564cb3ff299"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:16:39.365942 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.365932    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1465e850-eb74-4e35-9548-6564cb3ff299" (UID: "1465e850-eb74-4e35-9548-6564cb3ff299"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:16:39.366170 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.366144    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1465e850-eb74-4e35-9548-6564cb3ff299-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1465e850-eb74-4e35-9548-6564cb3ff299" (UID: "1465e850-eb74-4e35-9548-6564cb3ff299"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:16:39.366261 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.366172    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1465e850-eb74-4e35-9548-6564cb3ff299-kube-api-access-qp9cr" (OuterVolumeSpecName: "kube-api-access-qp9cr") pod "1465e850-eb74-4e35-9548-6564cb3ff299" (UID: "1465e850-eb74-4e35-9548-6564cb3ff299"). InnerVolumeSpecName "kube-api-access-qp9cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:16:39.418931 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.418899    2562 generic.go:356] "Generic (PLEG): container finished" podID="1465e850-eb74-4e35-9548-6564cb3ff299" containerID="5f837e7dfe903fe1fc37230e738a675753bcacb2e9875bd927afec2b5b79ac52" exitCode=0
Jul 27 20:16:39.419088 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.418946    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm" event={"ID":"1465e850-eb74-4e35-9548-6564cb3ff299","Type":"ContainerDied","Data":"5f837e7dfe903fe1fc37230e738a675753bcacb2e9875bd927afec2b5b79ac52"}
Jul 27 20:16:39.419088 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.418985    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm" event={"ID":"1465e850-eb74-4e35-9548-6564cb3ff299","Type":"ContainerDied","Data":"757f99eb77392a79221c7e935316f305fd5b72081a8d4757b6adbdc7034c440d"}
Jul 27 20:16:39.419088 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.418946    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"
Jul 27 20:16:39.419231 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.419006    2562 scope.go:117] "RemoveContainer" containerID="5f837e7dfe903fe1fc37230e738a675753bcacb2e9875bd927afec2b5b79ac52"
Jul 27 20:16:39.429129 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.429107    2562 scope.go:117] "RemoveContainer" containerID="266b84abf592301afffee3349991c839221b8ca539353a5e8b018d57956daf34"
Jul 27 20:16:39.439318 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.439269    2562 scope.go:117] "RemoveContainer" containerID="5f837e7dfe903fe1fc37230e738a675753bcacb2e9875bd927afec2b5b79ac52"
Jul 27 20:16:39.440272 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:16:39.440226    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f837e7dfe903fe1fc37230e738a675753bcacb2e9875bd927afec2b5b79ac52\": container with ID starting with 5f837e7dfe903fe1fc37230e738a675753bcacb2e9875bd927afec2b5b79ac52 not found: ID does not exist" containerID="5f837e7dfe903fe1fc37230e738a675753bcacb2e9875bd927afec2b5b79ac52"
Jul 27 20:16:39.440385 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.440267    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f837e7dfe903fe1fc37230e738a675753bcacb2e9875bd927afec2b5b79ac52"} err="failed to get container status \"5f837e7dfe903fe1fc37230e738a675753bcacb2e9875bd927afec2b5b79ac52\": rpc error: code = NotFound desc = could not find container \"5f837e7dfe903fe1fc37230e738a675753bcacb2e9875bd927afec2b5b79ac52\": container with ID starting with 5f837e7dfe903fe1fc37230e738a675753bcacb2e9875bd927afec2b5b79ac52 not found: ID does not exist"
Jul 27 20:16:39.440385 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.440297    2562 scope.go:117] "RemoveContainer" containerID="266b84abf592301afffee3349991c839221b8ca539353a5e8b018d57956daf34"
Jul 27 20:16:39.440674 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:16:39.440648    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"266b84abf592301afffee3349991c839221b8ca539353a5e8b018d57956daf34\": container with ID starting with 266b84abf592301afffee3349991c839221b8ca539353a5e8b018d57956daf34 not found: ID does not exist" containerID="266b84abf592301afffee3349991c839221b8ca539353a5e8b018d57956daf34"
Jul 27 20:16:39.440741 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.440677    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"266b84abf592301afffee3349991c839221b8ca539353a5e8b018d57956daf34"} err="failed to get container status \"266b84abf592301afffee3349991c839221b8ca539353a5e8b018d57956daf34\": rpc error: code = NotFound desc = could not find container \"266b84abf592301afffee3349991c839221b8ca539353a5e8b018d57956daf34\": container with ID starting with 266b84abf592301afffee3349991c839221b8ca539353a5e8b018d57956daf34 not found: ID does not exist"
Jul 27 20:16:39.442830 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.442812    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"]
Jul 27 20:16:39.446024 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.446002    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-849f6b-openshift-default-8547d8dbfb-7njdm"]
Jul 27 20:16:39.464439 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.464408    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1465e850-eb74-4e35-9548-6564cb3ff299-istiod-ca-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:39.464439 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.464432    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1465e850-eb74-4e35-9548-6564cb3ff299-istio-podinfo\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:39.464611 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.464448    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1465e850-eb74-4e35-9548-6564cb3ff299-istio-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:39.464611 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.464458    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-istio-data\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:39.464611 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.464472    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp9cr\" (UniqueName: \"kubernetes.io/projected/1465e850-eb74-4e35-9548-6564cb3ff299-kube-api-access-qp9cr\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:39.464611 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.464506    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1465e850-eb74-4e35-9548-6564cb3ff299-istio-envoy\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:39.992747 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:39.992714    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1465e850-eb74-4e35-9548-6564cb3ff299" path="/var/lib/kubelet/pods/1465e850-eb74-4e35-9548-6564cb3ff299/volumes"
Jul 27 20:16:44.903032 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:44.902994    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"]
Jul 27 20:16:44.903540 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:44.903315    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl" podUID="ce195c9f-1aad-4439-ba44-e6dffa27a6f7" containerName="maas-api" containerID="cri-o://e43fdcb680602f3a09cbb896f3a16007710c417f8c4ed8d6575c14ecf07759a5" gracePeriod=30
Jul 27 20:16:45.168280 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.168184    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:16:45.215707 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.215677    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-openshift-service-ca\") pod \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\" (UID: \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\") "
Jul 27 20:16:45.215901 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.215791    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zr85\" (UniqueName: \"kubernetes.io/projected/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-kube-api-access-7zr85\") pod \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\" (UID: \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\") "
Jul 27 20:16:45.215901 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.215827    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-maas-api-tls\") pod \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\" (UID: \"ce195c9f-1aad-4439-ba44-e6dffa27a6f7\") "
Jul 27 20:16:45.216128 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.216098    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ce195c9f-1aad-4439-ba44-e6dffa27a6f7" (UID: "ce195c9f-1aad-4439-ba44-e6dffa27a6f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:16:45.217940 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.217915    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-kube-api-access-7zr85" (OuterVolumeSpecName: "kube-api-access-7zr85") pod "ce195c9f-1aad-4439-ba44-e6dffa27a6f7" (UID: "ce195c9f-1aad-4439-ba44-e6dffa27a6f7"). InnerVolumeSpecName "kube-api-access-7zr85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:16:45.217940 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.217937    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ce195c9f-1aad-4439-ba44-e6dffa27a6f7" (UID: "ce195c9f-1aad-4439-ba44-e6dffa27a6f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:16:45.317279 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.317237    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zr85\" (UniqueName: \"kubernetes.io/projected/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-kube-api-access-7zr85\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:45.317279 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.317272    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:45.317279 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.317281    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce195c9f-1aad-4439-ba44-e6dffa27a6f7-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:16:45.451211 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.451119    2562 generic.go:356] "Generic (PLEG): container finished" podID="ce195c9f-1aad-4439-ba44-e6dffa27a6f7" containerID="e43fdcb680602f3a09cbb896f3a16007710c417f8c4ed8d6575c14ecf07759a5" exitCode=0
Jul 27 20:16:45.451211 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.451171    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"
Jul 27 20:16:45.451211 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.451198    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl" event={"ID":"ce195c9f-1aad-4439-ba44-e6dffa27a6f7","Type":"ContainerDied","Data":"e43fdcb680602f3a09cbb896f3a16007710c417f8c4ed8d6575c14ecf07759a5"}
Jul 27 20:16:45.451464 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.451258    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl" event={"ID":"ce195c9f-1aad-4439-ba44-e6dffa27a6f7","Type":"ContainerDied","Data":"c93e5ff15378c77ed0207175026c4a3fad430ccf57142bd9239f18af24ca1caa"}
Jul 27 20:16:45.451464 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.451281    2562 scope.go:117] "RemoveContainer" containerID="e43fdcb680602f3a09cbb896f3a16007710c417f8c4ed8d6575c14ecf07759a5"
Jul 27 20:16:45.460774 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.460755    2562 scope.go:117] "RemoveContainer" containerID="e43fdcb680602f3a09cbb896f3a16007710c417f8c4ed8d6575c14ecf07759a5"
Jul 27 20:16:45.461026 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:16:45.461008    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e43fdcb680602f3a09cbb896f3a16007710c417f8c4ed8d6575c14ecf07759a5\": container with ID starting with e43fdcb680602f3a09cbb896f3a16007710c417f8c4ed8d6575c14ecf07759a5 not found: ID does not exist" containerID="e43fdcb680602f3a09cbb896f3a16007710c417f8c4ed8d6575c14ecf07759a5"
Jul 27 20:16:45.461070 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.461035    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e43fdcb680602f3a09cbb896f3a16007710c417f8c4ed8d6575c14ecf07759a5"} err="failed to get container status \"e43fdcb680602f3a09cbb896f3a16007710c417f8c4ed8d6575c14ecf07759a5\": rpc error: code = NotFound desc = could not find container \"e43fdcb680602f3a09cbb896f3a16007710c417f8c4ed8d6575c14ecf07759a5\": container with ID starting with e43fdcb680602f3a09cbb896f3a16007710c417f8c4ed8d6575c14ecf07759a5 not found: ID does not exist"
Jul 27 20:16:45.472550 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.472461    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"]
Jul 27 20:16:45.477488 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.477457    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a5392-548c6b57fc-vhwhl"]
Jul 27 20:16:45.993111 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:45.993081    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce195c9f-1aad-4439-ba44-e6dffa27a6f7" path="/var/lib/kubelet/pods/ce195c9f-1aad-4439-ba44-e6dffa27a6f7/volumes"
Jul 27 20:16:56.185240 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:56.185198    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"]
Jul 27 20:16:56.185872 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:56.185459    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr" podUID="3a9262cb-5152-460c-bdbc-d65acb551c43" containerName="istio-proxy" containerID="cri-o://cf633e48e27220cc190c51d3dda271e64f2f9413c74a50105b135283007c5730" gracePeriod=30
Jul 27 20:16:57.070369 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.070335    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"]
Jul 27 20:16:57.070865 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.070846    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1465e850-eb74-4e35-9548-6564cb3ff299" containerName="istio-proxy"
Jul 27 20:16:57.070959 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.070868    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1465e850-eb74-4e35-9548-6564cb3ff299" containerName="istio-proxy"
Jul 27 20:16:57.070959 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.070895    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3dcd21-d7e7-49cf-8b65-010ada3208df" containerName="maas-api"
Jul 27 20:16:57.070959 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.070903    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3dcd21-d7e7-49cf-8b65-010ada3208df" containerName="maas-api"
Jul 27 20:16:57.070959 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.070913    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce195c9f-1aad-4439-ba44-e6dffa27a6f7" containerName="maas-api"
Jul 27 20:16:57.070959 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.070923    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce195c9f-1aad-4439-ba44-e6dffa27a6f7" containerName="maas-api"
Jul 27 20:16:57.070959 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.070935    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1465e850-eb74-4e35-9548-6564cb3ff299" containerName="istio-proxy"
Jul 27 20:16:57.070959 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.070943    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1465e850-eb74-4e35-9548-6564cb3ff299" containerName="istio-proxy"
Jul 27 20:16:57.071299 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.071049    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3dcd21-d7e7-49cf-8b65-010ada3208df" containerName="maas-api"
Jul 27 20:16:57.071299 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.071063    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce195c9f-1aad-4439-ba44-e6dffa27a6f7" containerName="maas-api"
Jul 27 20:16:57.071299 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.071075    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1465e850-eb74-4e35-9548-6564cb3ff299" containerName="istio-proxy"
Jul 27 20:16:57.071299 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.071086    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1465e850-eb74-4e35-9548-6564cb3ff299" containerName="istio-proxy"
Jul 27 20:16:57.074611 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.074588    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.078012 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.077991    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d453a8-openshift-default-dockercfg-b2svr\""
Jul 27 20:16:57.091118 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.091093    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"]
Jul 27 20:16:57.128762 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.128730    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-podinfo\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.128917 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.128767    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-workload-certs\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.128917 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.128795    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-envoy\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.128917 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.128826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-credential-socket\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.128917 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.128850    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-data\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.128917 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.128874    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-workload-socket\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.129088 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.128924    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-token\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.129088 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.128958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5247k\" (UniqueName: \"kubernetes.io/projected/d8d9f369-135c-4558-8086-8602d9d94f4c-kube-api-access-5247k\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.129088 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.128988    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d9f369-135c-4558-8086-8602d9d94f4c-istiod-ca-cert\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.229935 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.229902    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-podinfo\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.230376 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.229946    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-workload-certs\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.230376 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.229969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-envoy\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.230376 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.229991    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-credential-socket\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.230376 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.230018    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-data\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.230376 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.230049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-workload-socket\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.230376 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.230119    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-token\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.230376 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.230159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5247k\" (UniqueName: \"kubernetes.io/projected/d8d9f369-135c-4558-8086-8602d9d94f4c-kube-api-access-5247k\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.230376 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.230198    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d9f369-135c-4558-8086-8602d9d94f4c-istiod-ca-cert\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.230376 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.230354    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-workload-certs\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.230778 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.230429    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-data\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.230778 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.230461    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-credential-socket\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.230778 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.230632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-workload-socket\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.230965 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.230944    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d9f369-135c-4558-8086-8602d9d94f4c-istiod-ca-cert\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.232597 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.232580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-envoy\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.232597 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.232594    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-podinfo\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.239459 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.239432    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-token\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.239571 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.239513    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5247k\" (UniqueName: \"kubernetes.io/projected/d8d9f369-135c-4558-8086-8602d9d94f4c-kube-api-access-5247k\") pod \"e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") " pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.387801 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.387765    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:57.551308 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.551247    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"]
Jul 27 20:16:57.553893 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:16:57.553858    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d9f369_135c_4558_8086_8602d9d94f4c.slice/crio-127f03e119e84d7aa9ae71b8fe695f6733f173d0e9caca7180dadd7d7ce5bebd WatchSource:0}: Error finding container 127f03e119e84d7aa9ae71b8fe695f6733f173d0e9caca7180dadd7d7ce5bebd: Status 404 returned error can't find the container with id 127f03e119e84d7aa9ae71b8fe695f6733f173d0e9caca7180dadd7d7ce5bebd
Jul 27 20:16:57.556176 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.556142    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:16:57.556269 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.556223    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:16:57.556332 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:57.556270    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:16:58.522997 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:58.522961    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6" event={"ID":"d8d9f369-135c-4558-8086-8602d9d94f4c","Type":"ContainerStarted","Data":"e3ddd7d2f60f35d45742b713e8a6a8666ad8c825c2d38f07eaec75966f0f33cd"}
Jul 27 20:16:58.523388 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:58.523021    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6" event={"ID":"d8d9f369-135c-4558-8086-8602d9d94f4c","Type":"ContainerStarted","Data":"127f03e119e84d7aa9ae71b8fe695f6733f173d0e9caca7180dadd7d7ce5bebd"}
Jul 27 20:16:58.543838 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:58.543786    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6" podStartSLOduration=1.543767047 podStartE2EDuration="1.543767047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:16:58.542529331 +0000 UTC m=+2233.094445746" watchObservedRunningTime="2026-07-27 20:16:58.543767047 +0000 UTC m=+2233.095683523"
Jul 27 20:16:59.388737 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:59.388698    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:59.393598 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:59.393572    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:59.528565 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:59.528535    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:16:59.529702 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:16:59.529679    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:17:00.638138 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:00.638100    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"]
Jul 27 20:17:00.641822 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:00.641806    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:00.644121 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:00.644101    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d453a8-serving-cert\""
Jul 27 20:17:00.654147 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:00.654119    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"]
Jul 27 20:17:00.770910 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:00.770877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg9vk\" (UniqueName: \"kubernetes.io/projected/394a56f7-d2b8-4a37-9d08-1c277c4d5488-kube-api-access-wg9vk\") pod \"maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm\" (UID: \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:00.771132 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:00.770927    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/394a56f7-d2b8-4a37-9d08-1c277c4d5488-openshift-service-ca\") pod \"maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm\" (UID: \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:00.771132 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:00.771073    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/394a56f7-d2b8-4a37-9d08-1c277c4d5488-maas-api-tls\") pod \"maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm\" (UID: \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:00.872041 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:00.872008    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg9vk\" (UniqueName: \"kubernetes.io/projected/394a56f7-d2b8-4a37-9d08-1c277c4d5488-kube-api-access-wg9vk\") pod \"maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm\" (UID: \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:00.872212 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:00.872059    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/394a56f7-d2b8-4a37-9d08-1c277c4d5488-openshift-service-ca\") pod \"maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm\" (UID: \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:00.872212 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:00.872134    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/394a56f7-d2b8-4a37-9d08-1c277c4d5488-maas-api-tls\") pod \"maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm\" (UID: \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:00.872875 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:00.872848    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/394a56f7-d2b8-4a37-9d08-1c277c4d5488-openshift-service-ca\") pod \"maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm\" (UID: \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:00.874792 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:00.874764    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/394a56f7-d2b8-4a37-9d08-1c277c4d5488-maas-api-tls\") pod \"maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm\" (UID: \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:00.880417 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:00.880392    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg9vk\" (UniqueName: \"kubernetes.io/projected/394a56f7-d2b8-4a37-9d08-1c277c4d5488-kube-api-access-wg9vk\") pod \"maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm\" (UID: \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:00.954069 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:00.953980    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:01.118990 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.118965    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"]
Jul 27 20:17:01.120519 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:17:01.120468    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394a56f7_d2b8_4a37_9d08_1c277c4d5488.slice/crio-470b29f6e14130a3564947d7fd2e6d8da4b75da29a5cdcb8e467c03a1fcd8d86 WatchSource:0}: Error finding container 470b29f6e14130a3564947d7fd2e6d8da4b75da29a5cdcb8e467c03a1fcd8d86: Status 404 returned error can't find the container with id 470b29f6e14130a3564947d7fd2e6d8da4b75da29a5cdcb8e467c03a1fcd8d86
Jul 27 20:17:01.122242 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.122223    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:17:01.450138 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.450113    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:17:01.540584 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.540405    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm" event={"ID":"394a56f7-d2b8-4a37-9d08-1c277c4d5488","Type":"ContainerStarted","Data":"470b29f6e14130a3564947d7fd2e6d8da4b75da29a5cdcb8e467c03a1fcd8d86"}
Jul 27 20:17:01.542176 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.542149    2562 generic.go:356] "Generic (PLEG): container finished" podID="3a9262cb-5152-460c-bdbc-d65acb551c43" containerID="cf633e48e27220cc190c51d3dda271e64f2f9413c74a50105b135283007c5730" exitCode=0
Jul 27 20:17:01.542310 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.542256    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"
Jul 27 20:17:01.542310 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.542292    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr" event={"ID":"3a9262cb-5152-460c-bdbc-d65acb551c43","Type":"ContainerDied","Data":"cf633e48e27220cc190c51d3dda271e64f2f9413c74a50105b135283007c5730"}
Jul 27 20:17:01.542427 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.542351    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr" event={"ID":"3a9262cb-5152-460c-bdbc-d65acb551c43","Type":"ContainerDied","Data":"a6b664dd37b427583993293f712377613460256d8ae190030e018fbbca5c84e9"}
Jul 27 20:17:01.542427 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.542383    2562 scope.go:117] "RemoveContainer" containerID="cf633e48e27220cc190c51d3dda271e64f2f9413c74a50105b135283007c5730"
Jul 27 20:17:01.552102 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.552079    2562 scope.go:117] "RemoveContainer" containerID="5380a265c59111e3da2fddc9caec35775e32b8a9122adbc7dd82d9cca7b68d84"
Jul 27 20:17:01.562888 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.562871    2562 scope.go:117] "RemoveContainer" containerID="cf633e48e27220cc190c51d3dda271e64f2f9413c74a50105b135283007c5730"
Jul 27 20:17:01.563182 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:17:01.563162    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf633e48e27220cc190c51d3dda271e64f2f9413c74a50105b135283007c5730\": container with ID starting with cf633e48e27220cc190c51d3dda271e64f2f9413c74a50105b135283007c5730 not found: ID does not exist" containerID="cf633e48e27220cc190c51d3dda271e64f2f9413c74a50105b135283007c5730"
Jul 27 20:17:01.563267 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.563190    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf633e48e27220cc190c51d3dda271e64f2f9413c74a50105b135283007c5730"} err="failed to get container status \"cf633e48e27220cc190c51d3dda271e64f2f9413c74a50105b135283007c5730\": rpc error: code = NotFound desc = could not find container \"cf633e48e27220cc190c51d3dda271e64f2f9413c74a50105b135283007c5730\": container with ID starting with cf633e48e27220cc190c51d3dda271e64f2f9413c74a50105b135283007c5730 not found: ID does not exist"
Jul 27 20:17:01.563267 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.563208    2562 scope.go:117] "RemoveContainer" containerID="5380a265c59111e3da2fddc9caec35775e32b8a9122adbc7dd82d9cca7b68d84"
Jul 27 20:17:01.563452 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:17:01.563435    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5380a265c59111e3da2fddc9caec35775e32b8a9122adbc7dd82d9cca7b68d84\": container with ID starting with 5380a265c59111e3da2fddc9caec35775e32b8a9122adbc7dd82d9cca7b68d84 not found: ID does not exist" containerID="5380a265c59111e3da2fddc9caec35775e32b8a9122adbc7dd82d9cca7b68d84"
Jul 27 20:17:01.563521 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.563458    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5380a265c59111e3da2fddc9caec35775e32b8a9122adbc7dd82d9cca7b68d84"} err="failed to get container status \"5380a265c59111e3da2fddc9caec35775e32b8a9122adbc7dd82d9cca7b68d84\": rpc error: code = NotFound desc = could not find container \"5380a265c59111e3da2fddc9caec35775e32b8a9122adbc7dd82d9cca7b68d84\": container with ID starting with 5380a265c59111e3da2fddc9caec35775e32b8a9122adbc7dd82d9cca7b68d84 not found: ID does not exist"
Jul 27 20:17:01.581315 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.581276    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lpgt\" (UniqueName: \"kubernetes.io/projected/3a9262cb-5152-460c-bdbc-d65acb551c43-kube-api-access-2lpgt\") pod \"3a9262cb-5152-460c-bdbc-d65acb551c43\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") "
Jul 27 20:17:01.581509 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.581333    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-envoy\") pod \"3a9262cb-5152-460c-bdbc-d65acb551c43\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") "
Jul 27 20:17:01.581509 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.581387    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-podinfo\") pod \"3a9262cb-5152-460c-bdbc-d65acb551c43\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") "
Jul 27 20:17:01.581509 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.581419    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9262cb-5152-460c-bdbc-d65acb551c43-istiod-ca-cert\") pod \"3a9262cb-5152-460c-bdbc-d65acb551c43\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") "
Jul 27 20:17:01.581509 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.581443    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-credential-socket\") pod \"3a9262cb-5152-460c-bdbc-d65acb551c43\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") "
Jul 27 20:17:01.581761 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.581521    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-workload-socket\") pod \"3a9262cb-5152-460c-bdbc-d65acb551c43\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") "
Jul 27 20:17:01.581761 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.581554    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-data\") pod \"3a9262cb-5152-460c-bdbc-d65acb551c43\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") "
Jul 27 20:17:01.581761 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.581589    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-workload-certs\") pod \"3a9262cb-5152-460c-bdbc-d65acb551c43\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") "
Jul 27 20:17:01.581761 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.581628    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-token\") pod \"3a9262cb-5152-460c-bdbc-d65acb551c43\" (UID: \"3a9262cb-5152-460c-bdbc-d65acb551c43\") "
Jul 27 20:17:01.582705 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.582676    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9262cb-5152-460c-bdbc-d65acb551c43-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a9262cb-5152-460c-bdbc-d65acb551c43" (UID: "3a9262cb-5152-460c-bdbc-d65acb551c43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:17:01.582978 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.582869    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a9262cb-5152-460c-bdbc-d65acb551c43" (UID: "3a9262cb-5152-460c-bdbc-d65acb551c43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:17:01.583071 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.583033    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a9262cb-5152-460c-bdbc-d65acb551c43" (UID: "3a9262cb-5152-460c-bdbc-d65acb551c43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:17:01.583326 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.583306    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a9262cb-5152-460c-bdbc-d65acb551c43" (UID: "3a9262cb-5152-460c-bdbc-d65acb551c43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:17:01.583924 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.583869    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a9262cb-5152-460c-bdbc-d65acb551c43" (UID: "3a9262cb-5152-460c-bdbc-d65acb551c43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:17:01.585519 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.585491    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9262cb-5152-460c-bdbc-d65acb551c43-kube-api-access-2lpgt" (OuterVolumeSpecName: "kube-api-access-2lpgt") pod "3a9262cb-5152-460c-bdbc-d65acb551c43" (UID: "3a9262cb-5152-460c-bdbc-d65acb551c43"). InnerVolumeSpecName "kube-api-access-2lpgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:17:01.585645 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.585516    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a9262cb-5152-460c-bdbc-d65acb551c43" (UID: "3a9262cb-5152-460c-bdbc-d65acb551c43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:17:01.586264 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.586246    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a9262cb-5152-460c-bdbc-d65acb551c43" (UID: "3a9262cb-5152-460c-bdbc-d65acb551c43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:17:01.587143 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.587121    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a9262cb-5152-460c-bdbc-d65acb551c43" (UID: "3a9262cb-5152-460c-bdbc-d65acb551c43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:17:01.682641 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.682492    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-podinfo\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:01.682641 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.682525    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9262cb-5152-460c-bdbc-d65acb551c43-istiod-ca-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:01.682641 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.682538    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-credential-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:01.682641 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.682553    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-workload-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:01.682641 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.682568    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-data\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:01.682641 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.682579    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-workload-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:01.682641 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.682591    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:01.682641 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.682602    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lpgt\" (UniqueName: \"kubernetes.io/projected/3a9262cb-5152-460c-bdbc-d65acb551c43-kube-api-access-2lpgt\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:01.682641 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.682615    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9262cb-5152-460c-bdbc-d65acb551c43-istio-envoy\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:01.864153 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.864119    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"]
Jul 27 20:17:01.869078 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.869052    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0a5392-openshift-default-6bb698769b-gkhxr"]
Jul 27 20:17:01.991517 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:01.991465    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a9262cb-5152-460c-bdbc-d65acb551c43" path="/var/lib/kubelet/pods/3a9262cb-5152-460c-bdbc-d65acb551c43/volumes"
Jul 27 20:17:02.549958 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:02.549922    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm" event={"ID":"394a56f7-d2b8-4a37-9d08-1c277c4d5488","Type":"ContainerStarted","Data":"bc3e9adb2bb3dc1a4e96d035066a2da4a9744d7f5afb8d0dd671ad702c0cfd3c"}
Jul 27 20:17:02.550161 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:02.549984    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:02.566763 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:02.566698    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm" podStartSLOduration=1.959696061 podStartE2EDuration="2.566677686s" totalImagesPullingTime="606.981625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:17:02.566025644 +0000 UTC m=+2237.117942062" watchObservedRunningTime="2026-07-27 20:17:02.566677686 +0000 UTC m=+2237.118594105"
Jul 27 20:17:05.673654 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.673620    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"]
Jul 27 20:17:05.674282 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.674263    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9262cb-5152-460c-bdbc-d65acb551c43" containerName="istio-proxy"
Jul 27 20:17:05.674396 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.674284    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9262cb-5152-460c-bdbc-d65acb551c43" containerName="istio-proxy"
Jul 27 20:17:05.674458 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.674414    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a9262cb-5152-460c-bdbc-d65acb551c43" containerName="istio-proxy"
Jul 27 20:17:05.679258 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.679233    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.682746 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.682728    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-83e699-openshift-default-dockercfg-fvpmq\""
Jul 27 20:17:05.689851 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.689825    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"]
Jul 27 20:17:05.823653 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.823620    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-podinfo\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.823653 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.823654    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-workload-socket\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.823903 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.823683    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-workload-certs\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.823903 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.823742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-credential-socket\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.823903 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.823786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-data\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.823903 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.823823    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istiod-ca-cert\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.823903 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.823859    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xtjk\" (UniqueName: \"kubernetes.io/projected/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-kube-api-access-6xtjk\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.823903 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.823899    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-envoy\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.824120 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.823948    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-token\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.925320 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.925230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-podinfo\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.925320 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.925269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-workload-socket\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.925320 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.925296    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-workload-certs\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.925586 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.925332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-credential-socket\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.925586 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.925364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-data\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.925586 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.925398    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istiod-ca-cert\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.925586 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.925533    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xtjk\" (UniqueName: \"kubernetes.io/projected/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-kube-api-access-6xtjk\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.925790 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.925633    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-envoy\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.925790 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.925680    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-token\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.925790 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.925746    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-data\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.925949 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.925805    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-workload-socket\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.925949 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.925835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-credential-socket\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.925949 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.925891    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-workload-certs\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.926184 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.926133    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istiod-ca-cert\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.928014 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.927986    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-envoy\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.928162 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.927997    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-podinfo\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.934757 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.934732    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-token\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.934866 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.934854    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xtjk\" (UniqueName: \"kubernetes.io/projected/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-kube-api-access-6xtjk\") pod \"e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") " pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:05.991654 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:05.991625    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:06.139801 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:06.139775    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"]
Jul 27 20:17:06.141748 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:17:06.141709    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a631a3_9bba_4aa2_bcec_e35f50b5a39b.slice/crio-1aeca64c1e946dec1674001d9ca192d9c9805e3141185f408d9ffb97dd4966b3 WatchSource:0}: Error finding container 1aeca64c1e946dec1674001d9ca192d9c9805e3141185f408d9ffb97dd4966b3: Status 404 returned error can't find the container with id 1aeca64c1e946dec1674001d9ca192d9c9805e3141185f408d9ffb97dd4966b3
Jul 27 20:17:06.144275 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:06.144242    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:17:06.144356 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:06.144310    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:17:06.144356 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:06.144336    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:17:06.572420 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:06.572316    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps" event={"ID":"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b","Type":"ContainerStarted","Data":"78e0cb71ad7026d7a16237f5c18461bbcd07f9d4641401e71dcbc7aa0549191a"}
Jul 27 20:17:06.572420 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:06.572380    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps" event={"ID":"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b","Type":"ContainerStarted","Data":"1aeca64c1e946dec1674001d9ca192d9c9805e3141185f408d9ffb97dd4966b3"}
Jul 27 20:17:06.593473 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:06.593417    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps" podStartSLOduration=1.593402846 podStartE2EDuration="1.593402846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:17:06.591272707 +0000 UTC m=+2241.143189125" watchObservedRunningTime="2026-07-27 20:17:06.593402846 +0000 UTC m=+2241.145319263"
Jul 27 20:17:06.991786 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:06.991747    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:07.996981 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:07.996948    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:08.560925 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.560888    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:08.586939 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.586905    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:08.587841 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.587822    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:08.745374 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.745337    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"]
Jul 27 20:17:08.745963 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.745941    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9262cb-5152-460c-bdbc-d65acb551c43" containerName="istio-proxy"
Jul 27 20:17:08.745963 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.745964    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9262cb-5152-460c-bdbc-d65acb551c43" containerName="istio-proxy"
Jul 27 20:17:08.746171 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.746156    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a9262cb-5152-460c-bdbc-d65acb551c43" containerName="istio-proxy"
Jul 27 20:17:08.749847 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.749828    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:17:08.752189 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.752169    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-83e699-serving-cert\""
Jul 27 20:17:08.762033 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.762010    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"]
Jul 27 20:17:08.856777 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.856663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-maas-api-tls\") pod \"maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6\" (UID: \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:17:08.856777 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.856771    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6xl8\" (UniqueName: \"kubernetes.io/projected/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-kube-api-access-d6xl8\") pod \"maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6\" (UID: \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:17:08.857014 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.856872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-openshift-service-ca\") pod \"maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6\" (UID: \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:17:08.958157 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.958102    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-maas-api-tls\") pod \"maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6\" (UID: \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:17:08.958364 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.958170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6xl8\" (UniqueName: \"kubernetes.io/projected/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-kube-api-access-d6xl8\") pod \"maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6\" (UID: \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:17:08.958364 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.958249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-openshift-service-ca\") pod \"maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6\" (UID: \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:17:08.959119 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.959092    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-openshift-service-ca\") pod \"maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6\" (UID: \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:17:08.961255 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.961231    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-maas-api-tls\") pod \"maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6\" (UID: \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:17:08.966364 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:08.966343    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6xl8\" (UniqueName: \"kubernetes.io/projected/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-kube-api-access-d6xl8\") pod \"maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6\" (UID: \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:17:09.063067 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:09.063035    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:17:09.263937 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:09.263914    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"]
Jul 27 20:17:09.266060 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:17:09.266028    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d1ddf8_ac52_49a0_b3dd_104cf52fbc56.slice/crio-6331f3ff4618b22ee601ece709aac2179f692b8a7186e3be5d9401307753ec4b WatchSource:0}: Error finding container 6331f3ff4618b22ee601ece709aac2179f692b8a7186e3be5d9401307753ec4b: Status 404 returned error can't find the container with id 6331f3ff4618b22ee601ece709aac2179f692b8a7186e3be5d9401307753ec4b
Jul 27 20:17:09.599262 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:09.599168    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6" event={"ID":"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56","Type":"ContainerStarted","Data":"6331f3ff4618b22ee601ece709aac2179f692b8a7186e3be5d9401307753ec4b"}
Jul 27 20:17:10.608301 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:10.608256    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6" event={"ID":"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56","Type":"ContainerStarted","Data":"b8175fc1da5d3a358806887f3e33d621f6c923bd9d91ee45bed5fa357dcfa993"}
Jul 27 20:17:10.608766 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:10.608421    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:17:10.626271 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:10.626224    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6" podStartSLOduration=2.132870428 podStartE2EDuration="2.626209106s" totalImagesPullingTime="493.338678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:17:10.625715693 +0000 UTC m=+2245.177632137" watchObservedRunningTime="2026-07-27 20:17:10.626209106 +0000 UTC m=+2245.178125523"
Jul 27 20:17:15.570686 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:15.570639    2562 patch_prober.go:28] interesting pod/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 20:17:15.571158 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:15.570719    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6" podUID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 20:17:16.620133 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:16.620106    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:17:16.646464 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:16.646426    2562 generic.go:356] "Generic (PLEG): container finished" podID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerID="e3ddd7d2f60f35d45742b713e8a6a8666ad8c825c2d38f07eaec75966f0f33cd" exitCode=0
Jul 27 20:17:16.646713 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:16.646502    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6" event={"ID":"d8d9f369-135c-4558-8086-8602d9d94f4c","Type":"ContainerDied","Data":"e3ddd7d2f60f35d45742b713e8a6a8666ad8c825c2d38f07eaec75966f0f33cd"}
Jul 27 20:17:16.647014 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:16.646991    2562 scope.go:117] "RemoveContainer" containerID="e3ddd7d2f60f35d45742b713e8a6a8666ad8c825c2d38f07eaec75966f0f33cd"
Jul 27 20:17:16.648439 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:16.648389    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:17:16.648578 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:16.648518    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:17:16.648578 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:16.648570    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:17:17.655159 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:17.655110    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6" event={"ID":"d8d9f369-135c-4558-8086-8602d9d94f4c","Type":"ContainerStarted","Data":"936bb9dc914001ec36662434dd10b641764743ee5b8b2e659fe42612f2bde354"}
Jul 27 20:17:18.388567 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:18.388524    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:17:18.388896 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:18.388860    2562 patch_prober.go:28] interesting pod/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 27 20:17:18.388996 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:18.388950    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6" podUID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 27 20:17:19.388981 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:19.388935    2562 patch_prober.go:28] interesting pod/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 27 20:17:19.389499 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:19.389021    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6" podUID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 27 20:17:20.388593 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:20.388549    2562 patch_prober.go:28] interesting pod/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 27 20:17:20.388925 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:20.388895    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6" podUID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 27 20:17:21.388926 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:21.388888    2562 patch_prober.go:28] interesting pod/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 27 20:17:21.389546 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:21.389511    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6" podUID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 27 20:17:22.391778 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:22.391745    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:17:22.392280 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:22.392264    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:17:22.393244 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:22.393223    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:17:24.281890 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:24.281251    2562 patch_prober.go:28] interesting pod/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": read tcp 10.133.0.2:49798->10.133.0.70:15021: read: connection reset by peer" start-of-body=
Jul 27 20:17:24.281890 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:24.281344    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps" podUID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": read tcp 10.133.0.2:49798->10.133.0.70:15021: read: connection reset by peer"
Jul 27 20:17:24.696225 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:24.696189    2562 generic.go:356] "Generic (PLEG): container finished" podID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerID="78e0cb71ad7026d7a16237f5c18461bbcd07f9d4641401e71dcbc7aa0549191a" exitCode=0
Jul 27 20:17:24.696400 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:24.696262    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps" event={"ID":"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b","Type":"ContainerDied","Data":"78e0cb71ad7026d7a16237f5c18461bbcd07f9d4641401e71dcbc7aa0549191a"}
Jul 27 20:17:24.696815 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:24.696798    2562 scope.go:117] "RemoveContainer" containerID="78e0cb71ad7026d7a16237f5c18461bbcd07f9d4641401e71dcbc7aa0549191a"
Jul 27 20:17:24.697934 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:24.697905    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:17:24.697999 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:24.697975    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:17:24.698042 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:24.698005    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:17:25.706333 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:25.706297    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps" event={"ID":"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b","Type":"ContainerStarted","Data":"744ccd1fdd0980b6ca14cb139112e5e0341cdf0de48f4567b6beea99c0c79695"}
Jul 27 20:17:25.997907 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:25.997754    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:25.998502 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:25.998158    2562 patch_prober.go:28] interesting pod/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 27 20:17:25.998502 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:25.998210    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps" podUID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 27 20:17:26.992864 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:26.992818    2562 patch_prober.go:28] interesting pod/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 27 20:17:26.993420 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:26.992889    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps" podUID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 27 20:17:27.996906 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:27.996871    2562 patch_prober.go:28] interesting pod/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 27 20:17:27.997634 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:27.997603    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps" podUID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 27 20:17:28.993184 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:28.993144    2562 patch_prober.go:28] interesting pod/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 27 20:17:28.993911 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:28.993878    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps" podUID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 27 20:17:29.996120 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:29.996090    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:29.996541 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:29.996354    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:29.997015 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:29.997001    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:17:36.631371 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:36.631331    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"]
Jul 27 20:17:36.631966 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:36.631604    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm" podUID="394a56f7-d2b8-4a37-9d08-1c277c4d5488" containerName="maas-api" containerID="cri-o://bc3e9adb2bb3dc1a4e96d035066a2da4a9744d7f5afb8d0dd671ad702c0cfd3c" gracePeriod=30
Jul 27 20:17:36.766254 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:36.766218    2562 generic.go:356] "Generic (PLEG): container finished" podID="394a56f7-d2b8-4a37-9d08-1c277c4d5488" containerID="bc3e9adb2bb3dc1a4e96d035066a2da4a9744d7f5afb8d0dd671ad702c0cfd3c" exitCode=0
Jul 27 20:17:36.766407 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:36.766287    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm" event={"ID":"394a56f7-d2b8-4a37-9d08-1c277c4d5488","Type":"ContainerDied","Data":"bc3e9adb2bb3dc1a4e96d035066a2da4a9744d7f5afb8d0dd671ad702c0cfd3c"}
Jul 27 20:17:36.908816 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:36.908793    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:37.100325 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.100294    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg9vk\" (UniqueName: \"kubernetes.io/projected/394a56f7-d2b8-4a37-9d08-1c277c4d5488-kube-api-access-wg9vk\") pod \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\" (UID: \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\") "
Jul 27 20:17:37.100553 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.100368    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/394a56f7-d2b8-4a37-9d08-1c277c4d5488-openshift-service-ca\") pod \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\" (UID: \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\") "
Jul 27 20:17:37.100553 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.100459    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/394a56f7-d2b8-4a37-9d08-1c277c4d5488-maas-api-tls\") pod \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\" (UID: \"394a56f7-d2b8-4a37-9d08-1c277c4d5488\") "
Jul 27 20:17:37.100800 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.100776    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394a56f7-d2b8-4a37-9d08-1c277c4d5488-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "394a56f7-d2b8-4a37-9d08-1c277c4d5488" (UID: "394a56f7-d2b8-4a37-9d08-1c277c4d5488"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:17:37.102613 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.102588    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394a56f7-d2b8-4a37-9d08-1c277c4d5488-kube-api-access-wg9vk" (OuterVolumeSpecName: "kube-api-access-wg9vk") pod "394a56f7-d2b8-4a37-9d08-1c277c4d5488" (UID: "394a56f7-d2b8-4a37-9d08-1c277c4d5488"). InnerVolumeSpecName "kube-api-access-wg9vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:17:37.102738 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.102717    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394a56f7-d2b8-4a37-9d08-1c277c4d5488-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "394a56f7-d2b8-4a37-9d08-1c277c4d5488" (UID: "394a56f7-d2b8-4a37-9d08-1c277c4d5488"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:17:37.202058 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.201954    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/394a56f7-d2b8-4a37-9d08-1c277c4d5488-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:37.202058 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.201993    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/394a56f7-d2b8-4a37-9d08-1c277c4d5488-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:37.202058 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.202008    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg9vk\" (UniqueName: \"kubernetes.io/projected/394a56f7-d2b8-4a37-9d08-1c277c4d5488-kube-api-access-wg9vk\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:37.773918 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.773865    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm" event={"ID":"394a56f7-d2b8-4a37-9d08-1c277c4d5488","Type":"ContainerDied","Data":"470b29f6e14130a3564947d7fd2e6d8da4b75da29a5cdcb8e467c03a1fcd8d86"}
Jul 27 20:17:37.773918 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.773905    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"
Jul 27 20:17:37.774438 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.773937    2562 scope.go:117] "RemoveContainer" containerID="bc3e9adb2bb3dc1a4e96d035066a2da4a9744d7f5afb8d0dd671ad702c0cfd3c"
Jul 27 20:17:37.796106 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.796076    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"]
Jul 27 20:17:37.799805 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.799781    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d453a8-7b88469c5d-78qtm"]
Jul 27 20:17:37.992209 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:37.992171    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="394a56f7-d2b8-4a37-9d08-1c277c4d5488" path="/var/lib/kubelet/pods/394a56f7-d2b8-4a37-9d08-1c277c4d5488/volumes"
Jul 27 20:17:47.624841 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:47.624802    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"]
Jul 27 20:17:47.625322 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:47.625068    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6" podUID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerName="istio-proxy" containerID="cri-o://936bb9dc914001ec36662434dd10b641764743ee5b8b2e659fe42612f2bde354" gracePeriod=30
Jul 27 20:17:52.394451 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.394412    2562 patch_prober.go:28] interesting pod/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 20:17:52.394859 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.394502    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6" podUID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 20:17:52.861571 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.861534    2562 generic.go:356] "Generic (PLEG): container finished" podID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerID="936bb9dc914001ec36662434dd10b641764743ee5b8b2e659fe42612f2bde354" exitCode=0
Jul 27 20:17:52.861736 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.861603    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6" event={"ID":"d8d9f369-135c-4558-8086-8602d9d94f4c","Type":"ContainerDied","Data":"936bb9dc914001ec36662434dd10b641764743ee5b8b2e659fe42612f2bde354"}
Jul 27 20:17:52.861736 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.861664    2562 scope.go:117] "RemoveContainer" containerID="e3ddd7d2f60f35d45742b713e8a6a8666ad8c825c2d38f07eaec75966f0f33cd"
Jul 27 20:17:52.911371 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.911339    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:17:52.959973 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.959938    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-workload-certs\") pod \"d8d9f369-135c-4558-8086-8602d9d94f4c\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") "
Jul 27 20:17:52.959973 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.959969    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-workload-socket\") pod \"d8d9f369-135c-4558-8086-8602d9d94f4c\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") "
Jul 27 20:17:52.960224 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.960003    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-podinfo\") pod \"d8d9f369-135c-4558-8086-8602d9d94f4c\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") "
Jul 27 20:17:52.960224 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.960022    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-envoy\") pod \"d8d9f369-135c-4558-8086-8602d9d94f4c\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") "
Jul 27 20:17:52.960224 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.960106    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-token\") pod \"d8d9f369-135c-4558-8086-8602d9d94f4c\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") "
Jul 27 20:17:52.960224 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.960140    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-data\") pod \"d8d9f369-135c-4558-8086-8602d9d94f4c\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") "
Jul 27 20:17:52.960224 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.960184    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-credential-socket\") pod \"d8d9f369-135c-4558-8086-8602d9d94f4c\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") "
Jul 27 20:17:52.960224 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.960211    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5247k\" (UniqueName: \"kubernetes.io/projected/d8d9f369-135c-4558-8086-8602d9d94f4c-kube-api-access-5247k\") pod \"d8d9f369-135c-4558-8086-8602d9d94f4c\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") "
Jul 27 20:17:52.960588 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.960304    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d8d9f369-135c-4558-8086-8602d9d94f4c" (UID: "d8d9f369-135c-4558-8086-8602d9d94f4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:17:52.960588 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.960334    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d8d9f369-135c-4558-8086-8602d9d94f4c" (UID: "d8d9f369-135c-4558-8086-8602d9d94f4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:17:52.960588 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.960377    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d9f369-135c-4558-8086-8602d9d94f4c-istiod-ca-cert\") pod \"d8d9f369-135c-4558-8086-8602d9d94f4c\" (UID: \"d8d9f369-135c-4558-8086-8602d9d94f4c\") "
Jul 27 20:17:52.960588 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.960463    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d8d9f369-135c-4558-8086-8602d9d94f4c" (UID: "d8d9f369-135c-4558-8086-8602d9d94f4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:17:52.960849 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.960829    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-credential-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:52.960915 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.960854    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-workload-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:52.960915 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.960868    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-workload-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:52.961041 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.961013    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d9f369-135c-4558-8086-8602d9d94f4c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d8d9f369-135c-4558-8086-8602d9d94f4c" (UID: "d8d9f369-135c-4558-8086-8602d9d94f4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:17:52.961469 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.961440    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-data" (OuterVolumeSpecName: "istio-data") pod "d8d9f369-135c-4558-8086-8602d9d94f4c" (UID: "d8d9f369-135c-4558-8086-8602d9d94f4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:17:52.962921 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.962878    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d8d9f369-135c-4558-8086-8602d9d94f4c" (UID: "d8d9f369-135c-4558-8086-8602d9d94f4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:17:52.963174 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.963144    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d8d9f369-135c-4558-8086-8602d9d94f4c" (UID: "d8d9f369-135c-4558-8086-8602d9d94f4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:17:52.963281 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.963261    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d9f369-135c-4558-8086-8602d9d94f4c-kube-api-access-5247k" (OuterVolumeSpecName: "kube-api-access-5247k") pod "d8d9f369-135c-4558-8086-8602d9d94f4c" (UID: "d8d9f369-135c-4558-8086-8602d9d94f4c"). InnerVolumeSpecName "kube-api-access-5247k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:17:52.963330 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:52.963297    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d8d9f369-135c-4558-8086-8602d9d94f4c" (UID: "d8d9f369-135c-4558-8086-8602d9d94f4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:17:53.062110 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:53.062013    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d9f369-135c-4558-8086-8602d9d94f4c-istiod-ca-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:53.062110 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:53.062049    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-podinfo\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:53.062110 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:53.062063    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-envoy\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:53.062110 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:53.062076    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:53.062110 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:53.062089    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d9f369-135c-4558-8086-8602d9d94f4c-istio-data\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:53.062110 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:53.062100    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5247k\" (UniqueName: \"kubernetes.io/projected/d8d9f369-135c-4558-8086-8602d9d94f4c-kube-api-access-5247k\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:17:53.867843 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:53.867816    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"
Jul 27 20:17:53.868269 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:53.867851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6" event={"ID":"d8d9f369-135c-4558-8086-8602d9d94f4c","Type":"ContainerDied","Data":"127f03e119e84d7aa9ae71b8fe695f6733f173d0e9caca7180dadd7d7ce5bebd"}
Jul 27 20:17:53.868269 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:53.867905    2562 scope.go:117] "RemoveContainer" containerID="936bb9dc914001ec36662434dd10b641764743ee5b8b2e659fe42612f2bde354"
Jul 27 20:17:53.901060 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:53.901031    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"]
Jul 27 20:17:53.907071 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:53.907046    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d453a8-openshift-default-7787b6dd86-kw9j6"]
Jul 27 20:17:53.991068 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:17:53.991032    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8d9f369-135c-4558-8086-8602d9d94f4c" path="/var/lib/kubelet/pods/d8d9f369-135c-4558-8086-8602d9d94f4c/volumes"
Jul 27 20:18:03.420893 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.420853    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"]
Jul 27 20:18:03.421392 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.421065    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6" podUID="46d1ddf8-ac52-49a0-b3dd-104cf52fbc56" containerName="maas-api" containerID="cri-o://b8175fc1da5d3a358806887f3e33d621f6c923bd9d91ee45bed5fa357dcfa993" gracePeriod=30
Jul 27 20:18:03.687908 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.687849    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:18:03.764261 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.764233    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-maas-api-tls\") pod \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\" (UID: \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\") "
Jul 27 20:18:03.764426 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.764340    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6xl8\" (UniqueName: \"kubernetes.io/projected/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-kube-api-access-d6xl8\") pod \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\" (UID: \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\") "
Jul 27 20:18:03.764426 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.764400    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-openshift-service-ca\") pod \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\" (UID: \"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56\") "
Jul 27 20:18:03.764772 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.764748    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46d1ddf8-ac52-49a0-b3dd-104cf52fbc56" (UID: "46d1ddf8-ac52-49a0-b3dd-104cf52fbc56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:18:03.766605 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.766578    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46d1ddf8-ac52-49a0-b3dd-104cf52fbc56" (UID: "46d1ddf8-ac52-49a0-b3dd-104cf52fbc56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:18:03.766685 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.766652    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-kube-api-access-d6xl8" (OuterVolumeSpecName: "kube-api-access-d6xl8") pod "46d1ddf8-ac52-49a0-b3dd-104cf52fbc56" (UID: "46d1ddf8-ac52-49a0-b3dd-104cf52fbc56"). InnerVolumeSpecName "kube-api-access-d6xl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:18:03.866002 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.865957    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6xl8\" (UniqueName: \"kubernetes.io/projected/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-kube-api-access-d6xl8\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:18:03.866002 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.865991    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:18:03.866002 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.866000    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:18:03.922261 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.922229    2562 generic.go:356] "Generic (PLEG): container finished" podID="46d1ddf8-ac52-49a0-b3dd-104cf52fbc56" containerID="b8175fc1da5d3a358806887f3e33d621f6c923bd9d91ee45bed5fa357dcfa993" exitCode=0
Jul 27 20:18:03.922444 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.922275    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"
Jul 27 20:18:03.922444 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.922321    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6" event={"ID":"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56","Type":"ContainerDied","Data":"b8175fc1da5d3a358806887f3e33d621f6c923bd9d91ee45bed5fa357dcfa993"}
Jul 27 20:18:03.922444 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.922374    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6" event={"ID":"46d1ddf8-ac52-49a0-b3dd-104cf52fbc56","Type":"ContainerDied","Data":"6331f3ff4618b22ee601ece709aac2179f692b8a7186e3be5d9401307753ec4b"}
Jul 27 20:18:03.922444 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.922397    2562 scope.go:117] "RemoveContainer" containerID="b8175fc1da5d3a358806887f3e33d621f6c923bd9d91ee45bed5fa357dcfa993"
Jul 27 20:18:03.931995 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.931979    2562 scope.go:117] "RemoveContainer" containerID="b8175fc1da5d3a358806887f3e33d621f6c923bd9d91ee45bed5fa357dcfa993"
Jul 27 20:18:03.932256 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:18:03.932237    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8175fc1da5d3a358806887f3e33d621f6c923bd9d91ee45bed5fa357dcfa993\": container with ID starting with b8175fc1da5d3a358806887f3e33d621f6c923bd9d91ee45bed5fa357dcfa993 not found: ID does not exist" containerID="b8175fc1da5d3a358806887f3e33d621f6c923bd9d91ee45bed5fa357dcfa993"
Jul 27 20:18:03.932324 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.932264    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8175fc1da5d3a358806887f3e33d621f6c923bd9d91ee45bed5fa357dcfa993"} err="failed to get container status \"b8175fc1da5d3a358806887f3e33d621f6c923bd9d91ee45bed5fa357dcfa993\": rpc error: code = NotFound desc = could not find container \"b8175fc1da5d3a358806887f3e33d621f6c923bd9d91ee45bed5fa357dcfa993\": container with ID starting with b8175fc1da5d3a358806887f3e33d621f6c923bd9d91ee45bed5fa357dcfa993 not found: ID does not exist"
Jul 27 20:18:03.943920 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.943853    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"]
Jul 27 20:18:03.947608 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.947586    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-83e699-55b84d4d4-l5kg6"]
Jul 27 20:18:03.991637 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:03.991605    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46d1ddf8-ac52-49a0-b3dd-104cf52fbc56" path="/var/lib/kubelet/pods/46d1ddf8-ac52-49a0-b3dd-104cf52fbc56/volumes"
Jul 27 20:18:14.984200 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:14.984166    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"]
Jul 27 20:18:14.985174 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:14.985125    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps" podUID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerName="istio-proxy" containerID="cri-o://744ccd1fdd0980b6ca14cb139112e5e0341cdf0de48f4567b6beea99c0c79695" gracePeriod=30
Jul 27 20:18:14.999187 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:14.999153    2562 patch_prober.go:28] interesting pod/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 20:18:14.999331 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:14.999222    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps" podUID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 20:18:15.879003 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.878960    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"]
Jul 27 20:18:15.879632 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.879615    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerName="istio-proxy"
Jul 27 20:18:15.879685 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.879637    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerName="istio-proxy"
Jul 27 20:18:15.879685 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.879654    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="394a56f7-d2b8-4a37-9d08-1c277c4d5488" containerName="maas-api"
Jul 27 20:18:15.879685 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.879663    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="394a56f7-d2b8-4a37-9d08-1c277c4d5488" containerName="maas-api"
Jul 27 20:18:15.879787 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.879691    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46d1ddf8-ac52-49a0-b3dd-104cf52fbc56" containerName="maas-api"
Jul 27 20:18:15.879787 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.879700    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="46d1ddf8-ac52-49a0-b3dd-104cf52fbc56" containerName="maas-api"
Jul 27 20:18:15.879856 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.879790    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerName="istio-proxy"
Jul 27 20:18:15.879856 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.879805    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="46d1ddf8-ac52-49a0-b3dd-104cf52fbc56" containerName="maas-api"
Jul 27 20:18:15.879856 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.879813    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="394a56f7-d2b8-4a37-9d08-1c277c4d5488" containerName="maas-api"
Jul 27 20:18:15.884685 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.884662    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:15.886819 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.886800    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-c3a53e-openshift-default-dockercfg-7qxwj\""
Jul 27 20:18:15.898415 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.898393    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"]
Jul 27 20:18:15.987174 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.987132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-podinfo\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:15.987687 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.987198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-credential-socket\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:15.987687 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.987234    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-workload-certs\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:15.987687 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.987266    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-data\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:15.987687 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.987304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhr4t\" (UniqueName: \"kubernetes.io/projected/bf8d180a-511d-4e57-af8a-40be2ff7c277-kube-api-access-dhr4t\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:15.987687 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.987329    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-envoy\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:15.987687 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.987355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8d180a-511d-4e57-af8a-40be2ff7c277-istiod-ca-cert\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:15.987687 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.987391    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-workload-socket\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:15.987687 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:15.987415    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-token\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.088878 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.088838    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-workload-socket\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.088878 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.088880    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-token\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.089190 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.088937    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-podinfo\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.089190 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.089037    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-credential-socket\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.089190 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.089070    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-workload-certs\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.089190 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.089102    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-data\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.089190 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.089142    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhr4t\" (UniqueName: \"kubernetes.io/projected/bf8d180a-511d-4e57-af8a-40be2ff7c277-kube-api-access-dhr4t\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.089190 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.089173    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-envoy\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.089465 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.089207    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8d180a-511d-4e57-af8a-40be2ff7c277-istiod-ca-cert\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.089670 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.089644    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-workload-certs\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.089786 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.089677    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-workload-socket\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.089786 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.089698    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-credential-socket\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.089894 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.089806    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-data\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.090035 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.090003    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8d180a-511d-4e57-af8a-40be2ff7c277-istiod-ca-cert\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.091637 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.091616    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-envoy\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.091765 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.091745    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-podinfo\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.101366 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.101339    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-token\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.102288 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.102266    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhr4t\" (UniqueName: \"kubernetes.io/projected/bf8d180a-511d-4e57-af8a-40be2ff7c277-kube-api-access-dhr4t\") pod \"e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") " pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.195586 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.195472    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:16.351988 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.351962    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"]
Jul 27 20:18:16.354546 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:18:16.354522    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8d180a_511d_4e57_af8a_40be2ff7c277.slice/crio-a842ef623077658240c3de78c4d21c1f68bd4b134639bd4b86d8f1efead0b241 WatchSource:0}: Error finding container a842ef623077658240c3de78c4d21c1f68bd4b134639bd4b86d8f1efead0b241: Status 404 returned error can't find the container with id a842ef623077658240c3de78c4d21c1f68bd4b134639bd4b86d8f1efead0b241
Jul 27 20:18:16.356716 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.356685    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:18:16.356792 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.356754    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:18:16.356848 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:16.356791    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:18:17.001666 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:17.001619    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6" event={"ID":"bf8d180a-511d-4e57-af8a-40be2ff7c277","Type":"ContainerStarted","Data":"bc25defe80a660f8ad34cf1e0ff1abab5e06213d2610ac6aae85c4ba5741ede2"}
Jul 27 20:18:17.002137 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:17.001679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6" event={"ID":"bf8d180a-511d-4e57-af8a-40be2ff7c277","Type":"ContainerStarted","Data":"a842ef623077658240c3de78c4d21c1f68bd4b134639bd4b86d8f1efead0b241"}
Jul 27 20:18:17.024676 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:17.024620    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6" podStartSLOduration=2.02460322 podStartE2EDuration="2.02460322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:17.022794321 +0000 UTC m=+2311.574710740" watchObservedRunningTime="2026-07-27 20:18:17.02460322 +0000 UTC m=+2311.576519637"
Jul 27 20:18:17.195669 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:17.195634    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:17.200824 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:17.200798    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:18.006429 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:18.006397    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:18.007431 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:18.007410    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:19.136778 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.136741    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"]
Jul 27 20:18:19.137385 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.137369    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerName="istio-proxy"
Jul 27 20:18:19.137457 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.137387    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerName="istio-proxy"
Jul 27 20:18:19.137539 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.137526    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8d9f369-135c-4558-8086-8602d9d94f4c" containerName="istio-proxy"
Jul 27 20:18:19.141131 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.141108    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:18:19.143364 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.143342    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-c3a53e-serving-cert\""
Jul 27 20:18:19.150199 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.149964    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"]
Jul 27 20:18:19.218026 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.217989    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb4fh\" (UniqueName: \"kubernetes.io/projected/301637fc-59a2-422b-a0d1-95ca7027f450-kube-api-access-vb4fh\") pod \"maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq\" (UID: \"301637fc-59a2-422b-a0d1-95ca7027f450\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:18:19.218200 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.218062    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/301637fc-59a2-422b-a0d1-95ca7027f450-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq\" (UID: \"301637fc-59a2-422b-a0d1-95ca7027f450\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:18:19.218200 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.218120    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/301637fc-59a2-422b-a0d1-95ca7027f450-maas-api-tls\") pod \"maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq\" (UID: \"301637fc-59a2-422b-a0d1-95ca7027f450\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:18:19.319617 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.319575    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/301637fc-59a2-422b-a0d1-95ca7027f450-maas-api-tls\") pod \"maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq\" (UID: \"301637fc-59a2-422b-a0d1-95ca7027f450\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:18:19.319791 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.319640    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb4fh\" (UniqueName: \"kubernetes.io/projected/301637fc-59a2-422b-a0d1-95ca7027f450-kube-api-access-vb4fh\") pod \"maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq\" (UID: \"301637fc-59a2-422b-a0d1-95ca7027f450\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:18:19.319791 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.319717    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/301637fc-59a2-422b-a0d1-95ca7027f450-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq\" (UID: \"301637fc-59a2-422b-a0d1-95ca7027f450\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:18:19.319791 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:18:19.319721    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-serving-cert: secret "maas-api-e2e-shared-a-c3a53e-serving-cert" not found
Jul 27 20:18:19.319791 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:18:19.319785    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301637fc-59a2-422b-a0d1-95ca7027f450-maas-api-tls podName:301637fc-59a2-422b-a0d1-95ca7027f450 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:19.81976891 +0000 UTC m=+2314.371685311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/301637fc-59a2-422b-a0d1-95ca7027f450-maas-api-tls") pod "maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq" (UID: "301637fc-59a2-422b-a0d1-95ca7027f450") : secret "maas-api-e2e-shared-a-c3a53e-serving-cert" not found
Jul 27 20:18:19.320456 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.320434    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/301637fc-59a2-422b-a0d1-95ca7027f450-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq\" (UID: \"301637fc-59a2-422b-a0d1-95ca7027f450\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:18:19.328527 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.328471    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb4fh\" (UniqueName: \"kubernetes.io/projected/301637fc-59a2-422b-a0d1-95ca7027f450-kube-api-access-vb4fh\") pod \"maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq\" (UID: \"301637fc-59a2-422b-a0d1-95ca7027f450\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:18:19.827292 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.827254    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/301637fc-59a2-422b-a0d1-95ca7027f450-maas-api-tls\") pod \"maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq\" (UID: \"301637fc-59a2-422b-a0d1-95ca7027f450\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:18:19.829863 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:19.829833    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/301637fc-59a2-422b-a0d1-95ca7027f450-maas-api-tls\") pod \"maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq\" (UID: \"301637fc-59a2-422b-a0d1-95ca7027f450\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:18:20.054983 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.054950    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:18:20.265897 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.265875    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:18:20.333308 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.333270    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-envoy\") pod \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") "
Jul 27 20:18:20.333490 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.333329    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-podinfo\") pod \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") "
Jul 27 20:18:20.333490 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.333357    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-credential-socket\") pod \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") "
Jul 27 20:18:20.333490 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.333403    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-token\") pod \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") "
Jul 27 20:18:20.333490 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.333425    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-data\") pod \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") "
Jul 27 20:18:20.333490 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.333447    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-workload-certs\") pod \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") "
Jul 27 20:18:20.333737 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.333530    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-workload-socket\") pod \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") "
Jul 27 20:18:20.333737 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.333579    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xtjk\" (UniqueName: \"kubernetes.io/projected/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-kube-api-access-6xtjk\") pod \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") "
Jul 27 20:18:20.333737 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.333612    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istiod-ca-cert\") pod \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\" (UID: \"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b\") "
Jul 27 20:18:20.333893 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.333773    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" (UID: "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:18:20.333949 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.333907    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" (UID: "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:18:20.334063 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.333990    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-credential-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:18:20.334240 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.334212    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" (UID: "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:18:20.334364 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.334290    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" (UID: "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:18:20.334809 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.334774    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-data" (OuterVolumeSpecName: "istio-data") pod "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" (UID: "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:18:20.336170 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.336148    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-token" (OuterVolumeSpecName: "istio-token") pod "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" (UID: "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:18:20.336170 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.336160    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" (UID: "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:18:20.336289 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.336264    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" (UID: "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:18:20.336503 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.336465    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-kube-api-access-6xtjk" (OuterVolumeSpecName: "kube-api-access-6xtjk") pod "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" (UID: "a8a631a3-9bba-4aa2-bcec-e35f50b5a39b"). InnerVolumeSpecName "kube-api-access-6xtjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:18:20.414891 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.414861    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"]
Jul 27 20:18:20.417156 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:18:20.417123    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301637fc_59a2_422b_a0d1_95ca7027f450.slice/crio-ea1866ef24c33faf6bf7ce478d5c08a321a3e9afa5325217243bcae39774bcd8 WatchSource:0}: Error finding container ea1866ef24c33faf6bf7ce478d5c08a321a3e9afa5325217243bcae39774bcd8: Status 404 returned error can't find the container with id ea1866ef24c33faf6bf7ce478d5c08a321a3e9afa5325217243bcae39774bcd8
Jul 27 20:18:20.434686 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.434660    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-workload-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:18:20.434686 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.434684    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xtjk\" (UniqueName: \"kubernetes.io/projected/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-kube-api-access-6xtjk\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:18:20.434850 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.434695    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istiod-ca-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:18:20.434850 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.434704    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-envoy\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:18:20.434850 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.434713    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-podinfo\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:18:20.434850 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.434720    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:18:20.434850 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.434728    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-istio-data\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:18:20.434850 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:20.434735    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b-workload-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:18:21.023460 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.023420    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq" event={"ID":"301637fc-59a2-422b-a0d1-95ca7027f450","Type":"ContainerStarted","Data":"9e27b47efe29d6b88482a5f9120ffd9094be73f5421c6d62f2a936957110f710"}
Jul 27 20:18:21.023702 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.023501    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq" event={"ID":"301637fc-59a2-422b-a0d1-95ca7027f450","Type":"ContainerStarted","Data":"ea1866ef24c33faf6bf7ce478d5c08a321a3e9afa5325217243bcae39774bcd8"}
Jul 27 20:18:21.023702 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.023536    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:18:21.025204 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.025180    2562 generic.go:356] "Generic (PLEG): container finished" podID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerID="744ccd1fdd0980b6ca14cb139112e5e0341cdf0de48f4567b6beea99c0c79695" exitCode=0
Jul 27 20:18:21.025339 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.025223    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"
Jul 27 20:18:21.025339 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.025260    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps" event={"ID":"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b","Type":"ContainerDied","Data":"744ccd1fdd0980b6ca14cb139112e5e0341cdf0de48f4567b6beea99c0c79695"}
Jul 27 20:18:21.025339 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.025305    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps" event={"ID":"a8a631a3-9bba-4aa2-bcec-e35f50b5a39b","Type":"ContainerDied","Data":"1aeca64c1e946dec1674001d9ca192d9c9805e3141185f408d9ffb97dd4966b3"}
Jul 27 20:18:21.025339 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.025333    2562 scope.go:117] "RemoveContainer" containerID="744ccd1fdd0980b6ca14cb139112e5e0341cdf0de48f4567b6beea99c0c79695"
Jul 27 20:18:21.035076 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.035058    2562 scope.go:117] "RemoveContainer" containerID="78e0cb71ad7026d7a16237f5c18461bbcd07f9d4641401e71dcbc7aa0549191a"
Jul 27 20:18:21.042991 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.042944    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq" podStartSLOduration=1.591208797 podStartE2EDuration="2.042928026s" totalImagesPullingTime="451.719229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:21.040027934 +0000 UTC m=+2315.591944363" watchObservedRunningTime="2026-07-27 20:18:21.042928026 +0000 UTC m=+2315.594844444"
Jul 27 20:18:21.043706 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.043687    2562 scope.go:117] "RemoveContainer" containerID="744ccd1fdd0980b6ca14cb139112e5e0341cdf0de48f4567b6beea99c0c79695"
Jul 27 20:18:21.044000 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:18:21.043981    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744ccd1fdd0980b6ca14cb139112e5e0341cdf0de48f4567b6beea99c0c79695\": container with ID starting with 744ccd1fdd0980b6ca14cb139112e5e0341cdf0de48f4567b6beea99c0c79695 not found: ID does not exist" containerID="744ccd1fdd0980b6ca14cb139112e5e0341cdf0de48f4567b6beea99c0c79695"
Jul 27 20:18:21.044070 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.044009    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744ccd1fdd0980b6ca14cb139112e5e0341cdf0de48f4567b6beea99c0c79695"} err="failed to get container status \"744ccd1fdd0980b6ca14cb139112e5e0341cdf0de48f4567b6beea99c0c79695\": rpc error: code = NotFound desc = could not find container \"744ccd1fdd0980b6ca14cb139112e5e0341cdf0de48f4567b6beea99c0c79695\": container with ID starting with 744ccd1fdd0980b6ca14cb139112e5e0341cdf0de48f4567b6beea99c0c79695 not found: ID does not exist"
Jul 27 20:18:21.044070 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.044027    2562 scope.go:117] "RemoveContainer" containerID="78e0cb71ad7026d7a16237f5c18461bbcd07f9d4641401e71dcbc7aa0549191a"
Jul 27 20:18:21.044283 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:18:21.044265    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e0cb71ad7026d7a16237f5c18461bbcd07f9d4641401e71dcbc7aa0549191a\": container with ID starting with 78e0cb71ad7026d7a16237f5c18461bbcd07f9d4641401e71dcbc7aa0549191a not found: ID does not exist" containerID="78e0cb71ad7026d7a16237f5c18461bbcd07f9d4641401e71dcbc7aa0549191a"
Jul 27 20:18:21.044324 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.044289    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e0cb71ad7026d7a16237f5c18461bbcd07f9d4641401e71dcbc7aa0549191a"} err="failed to get container status \"78e0cb71ad7026d7a16237f5c18461bbcd07f9d4641401e71dcbc7aa0549191a\": rpc error: code = NotFound desc = could not find container \"78e0cb71ad7026d7a16237f5c18461bbcd07f9d4641401e71dcbc7aa0549191a\": container with ID starting with 78e0cb71ad7026d7a16237f5c18461bbcd07f9d4641401e71dcbc7aa0549191a not found: ID does not exist"
Jul 27 20:18:21.055392 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.055369    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"]
Jul 27 20:18:21.059159 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.059136    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-83e699-openshift-default-5d69dc9df4-77kps"]
Jul 27 20:18:21.992291 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:21.992256    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" path="/var/lib/kubelet/pods/a8a631a3-9bba-4aa2-bcec-e35f50b5a39b/volumes"
Jul 27 20:18:24.005077 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.005040    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"]
Jul 27 20:18:24.006451 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.006420    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerName="istio-proxy"
Jul 27 20:18:24.006451 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.006444    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerName="istio-proxy"
Jul 27 20:18:24.006674 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.006549    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerName="istio-proxy"
Jul 27 20:18:24.011682 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.011653    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"]
Jul 27 20:18:24.011806 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.011754    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.014563 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.014540    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-f44149-openshift-default-dockercfg-gq9bv\""
Jul 27 20:18:24.073338 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.072881    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-workload-certs\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.073338 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.072950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-credential-socket\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.073338 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.072975    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-data\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.073338 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.073033    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eb244db-e1bc-4802-9e2a-69a5de755d51-istiod-ca-cert\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.073338 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.073219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pngn\" (UniqueName: \"kubernetes.io/projected/2eb244db-e1bc-4802-9e2a-69a5de755d51-kube-api-access-7pngn\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.073338 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.073294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-workload-socket\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.073824 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.073372    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-envoy\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.073824 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.073713    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-podinfo\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.074507 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.073941    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-token\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.174906 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.174859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eb244db-e1bc-4802-9e2a-69a5de755d51-istiod-ca-cert\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.175110 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.174927    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pngn\" (UniqueName: \"kubernetes.io/projected/2eb244db-e1bc-4802-9e2a-69a5de755d51-kube-api-access-7pngn\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.175110 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.174963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-workload-socket\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.175110 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.174985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-envoy\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.175110 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.175032    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-podinfo\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.175110 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.175095    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-token\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.175381 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.175135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-workload-certs\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.175381 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.175167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-credential-socket\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.175381 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.175194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-data\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.175752 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.175667    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-data\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.176038 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.175816    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-credential-socket\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.176038 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.175836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-workload-certs\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.176164 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.176107    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eb244db-e1bc-4802-9e2a-69a5de755d51-istiod-ca-cert\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.176164 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.176148    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-workload-socket\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.177649 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.177621    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-envoy\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.178190 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.178164    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-podinfo\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.184965 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.184938    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-token\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.185303 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.185276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pngn\" (UniqueName: \"kubernetes.io/projected/2eb244db-e1bc-4802-9e2a-69a5de755d51-kube-api-access-7pngn\") pod \"e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") " pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.325665 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.325573    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:24.476934 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.476893    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"]
Jul 27 20:18:24.479925 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:18:24.479892    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb244db_e1bc_4802_9e2a_69a5de755d51.slice/crio-d602461236875b14c8773c9fd0c31f025759c24c9142892ec189b83a93ed10f0 WatchSource:0}: Error finding container d602461236875b14c8773c9fd0c31f025759c24c9142892ec189b83a93ed10f0: Status 404 returned error can't find the container with id d602461236875b14c8773c9fd0c31f025759c24c9142892ec189b83a93ed10f0
Jul 27 20:18:24.482392 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.482360    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:18:24.482471 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.482431    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:18:24.482471 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:24.482459    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:18:25.050415 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:25.050381    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw" event={"ID":"2eb244db-e1bc-4802-9e2a-69a5de755d51","Type":"ContainerStarted","Data":"062e69e32e0fee38acdae45c2f9f9024da3a632e3f70eb6150384992c85776bb"}
Jul 27 20:18:25.050812 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:25.050431    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw" event={"ID":"2eb244db-e1bc-4802-9e2a-69a5de755d51","Type":"ContainerStarted","Data":"d602461236875b14c8773c9fd0c31f025759c24c9142892ec189b83a93ed10f0"}
Jul 27 20:18:25.072464 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:25.072404    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw" podStartSLOduration=2.072388494 podStartE2EDuration="2.072388494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:25.07004591 +0000 UTC m=+2319.621962324" watchObservedRunningTime="2026-07-27 20:18:25.072388494 +0000 UTC m=+2319.624304911"
Jul 27 20:18:25.326456 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:25.326365    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:25.331588 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:25.331564    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:26.055432 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:26.055397    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:26.056321 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:26.056303    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:27.037662 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.037620    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:18:27.305895 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.305817    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"]
Jul 27 20:18:27.306303 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.306290    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerName="istio-proxy"
Jul 27 20:18:27.306359 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.306307    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerName="istio-proxy"
Jul 27 20:18:27.306409 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.306396    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8a631a3-9bba-4aa2-bcec-e35f50b5a39b" containerName="istio-proxy"
Jul 27 20:18:27.309821 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.309794    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:18:27.312184 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.312157    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-f44149-serving-cert\""
Jul 27 20:18:27.324396 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.324367    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"]
Jul 27 20:18:27.409518 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.409459    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-maas-api-tls\") pod \"maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk\" (UID: \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:18:27.409810 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.409791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9rg\" (UniqueName: \"kubernetes.io/projected/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-kube-api-access-hf9rg\") pod \"maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk\" (UID: \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:18:27.409924 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.409911    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk\" (UID: \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:18:27.511397 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.511364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-maas-api-tls\") pod \"maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk\" (UID: \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:18:27.511397 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.511414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9rg\" (UniqueName: \"kubernetes.io/projected/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-kube-api-access-hf9rg\") pod \"maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk\" (UID: \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:18:27.511700 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.511446    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk\" (UID: \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:18:27.511700 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:18:27.511536    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-serving-cert: secret "maas-api-e2e-shared-b-f44149-serving-cert" not found
Jul 27 20:18:27.511700 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:18:27.511600    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-maas-api-tls podName:0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc nodeName:}" failed. No retries permitted until 2026-07-27 20:18:28.01158239 +0000 UTC m=+2322.563498790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-maas-api-tls") pod "maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk" (UID: "0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc") : secret "maas-api-e2e-shared-b-f44149-serving-cert" not found
Jul 27 20:18:27.512146 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.512129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk\" (UID: \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:18:27.526590 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:27.526562    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9rg\" (UniqueName: \"kubernetes.io/projected/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-kube-api-access-hf9rg\") pod \"maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk\" (UID: \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:18:28.016031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:28.015999    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-maas-api-tls\") pod \"maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk\" (UID: \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:18:28.019051 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:28.019018    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-maas-api-tls\") pod \"maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk\" (UID: \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:18:28.221447 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:28.221398    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:18:28.388160 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:28.387864    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"]
Jul 27 20:18:28.390515 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:18:28.390467    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da2b52f_fc77_4e9f_8be3_3d45bfeaedcc.slice/crio-342b1e00b0c2bbba3a8e7b7dc406bb26d4a64af755f3511a7cfa6d01141ad970 WatchSource:0}: Error finding container 342b1e00b0c2bbba3a8e7b7dc406bb26d4a64af755f3511a7cfa6d01141ad970: Status 404 returned error can't find the container with id 342b1e00b0c2bbba3a8e7b7dc406bb26d4a64af755f3511a7cfa6d01141ad970
Jul 27 20:18:29.073111 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:29.072734    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk" event={"ID":"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc","Type":"ContainerStarted","Data":"ad81419c38f8d58fb62f3d44c95eb4ecbe1cc71e95b1f93a8b621b62ad34eece"}
Jul 27 20:18:29.073111 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:29.072794    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk" event={"ID":"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc","Type":"ContainerStarted","Data":"342b1e00b0c2bbba3a8e7b7dc406bb26d4a64af755f3511a7cfa6d01141ad970"}
Jul 27 20:18:29.073111 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:29.073011    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:18:34.047749 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:34.047697    2562 patch_prober.go:28] interesting pod/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 20:18:34.048238 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:34.047813    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6" podUID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 20:18:35.088325 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:35.088295    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:18:35.106667 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:35.106598    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk" podStartSLOduration=7.801504936 podStartE2EDuration="8.106576076s" totalImagesPullingTime="305.07114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:29.094437869 +0000 UTC m=+2323.646354287" watchObservedRunningTime="2026-07-27 20:18:35.106576076 +0000 UTC m=+2329.658492495"
Jul 27 20:18:36.134490 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:36.134442    2562 generic.go:356] "Generic (PLEG): container finished" podID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerID="bc25defe80a660f8ad34cf1e0ff1abab5e06213d2610ac6aae85c4ba5741ede2" exitCode=0
Jul 27 20:18:36.134913 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:36.134521    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6" event={"ID":"bf8d180a-511d-4e57-af8a-40be2ff7c277","Type":"ContainerDied","Data":"bc25defe80a660f8ad34cf1e0ff1abab5e06213d2610ac6aae85c4ba5741ede2"}
Jul 27 20:18:36.134959 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:36.134918    2562 scope.go:117] "RemoveContainer" containerID="bc25defe80a660f8ad34cf1e0ff1abab5e06213d2610ac6aae85c4ba5741ede2"
Jul 27 20:18:36.136024 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:36.135998    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:18:36.136108 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:36.136064    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:18:36.136108 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:36.136095    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:18:37.142308 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:37.142228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6" event={"ID":"bf8d180a-511d-4e57-af8a-40be2ff7c277","Type":"ContainerStarted","Data":"b47f1097c562a899be22823e2b637d8078e550f7dd5c05ddd31bf58084684a3a"}
Jul 27 20:18:37.196729 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:37.196694    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:37.197009 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:37.196763    2562 patch_prober.go:28] interesting pod/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Jul 27 20:18:37.197171 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:37.197148    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6" podUID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Jul 27 20:18:38.196723 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:38.196685    2562 patch_prober.go:28] interesting pod/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Jul 27 20:18:38.197286 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:38.197253    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6" podUID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Jul 27 20:18:39.197303 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:39.196856    2562 patch_prober.go:28] interesting pod/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Jul 27 20:18:39.197303 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:39.196921    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6" podUID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Jul 27 20:18:40.196620 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:40.196579    2562 patch_prober.go:28] interesting pod/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Jul 27 20:18:40.196928 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:40.196897    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6" podUID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Jul 27 20:18:41.196689 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:41.196643    2562 patch_prober.go:28] interesting pod/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Jul 27 20:18:41.197161 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:41.196721    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6" podUID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Jul 27 20:18:42.200038 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:42.200005    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:42.200515 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:42.200504    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:42.201234 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:42.201210    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:18:45.191977 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:45.191939    2562 generic.go:356] "Generic (PLEG): container finished" podID="2eb244db-e1bc-4802-9e2a-69a5de755d51" containerID="062e69e32e0fee38acdae45c2f9f9024da3a632e3f70eb6150384992c85776bb" exitCode=0
Jul 27 20:18:45.192366 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:45.192006    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw" event={"ID":"2eb244db-e1bc-4802-9e2a-69a5de755d51","Type":"ContainerDied","Data":"062e69e32e0fee38acdae45c2f9f9024da3a632e3f70eb6150384992c85776bb"}
Jul 27 20:18:45.192453 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:45.192438    2562 scope.go:117] "RemoveContainer" containerID="062e69e32e0fee38acdae45c2f9f9024da3a632e3f70eb6150384992c85776bb"
Jul 27 20:18:45.193530 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:45.193499    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:18:45.193626 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:45.193577    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:18:45.193626 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:45.193611    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:18:46.202696 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:46.202646    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw" event={"ID":"2eb244db-e1bc-4802-9e2a-69a5de755d51","Type":"ContainerStarted","Data":"c45d6fba40a2f9f4efcae25719f559d82fbbdf06fea4351f5f167fd08908ff6a"}
Jul 27 20:18:46.326799 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:46.326767    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:46.326984 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:46.326961    2562 patch_prober.go:28] interesting pod/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Jul 27 20:18:46.327036 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:46.327014    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw" podUID="2eb244db-e1bc-4802-9e2a-69a5de755d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Jul 27 20:18:47.326586 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:47.326548    2562 patch_prober.go:28] interesting pod/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Jul 27 20:18:47.326977 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:47.326602    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw" podUID="2eb244db-e1bc-4802-9e2a-69a5de755d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Jul 27 20:18:48.326932 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:48.326892    2562 patch_prober.go:28] interesting pod/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Jul 27 20:18:48.327332 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:48.326963    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw" podUID="2eb244db-e1bc-4802-9e2a-69a5de755d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Jul 27 20:18:49.325974 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:49.325937    2562 patch_prober.go:28] interesting pod/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Jul 27 20:18:49.326153 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:49.325993    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw" podUID="2eb244db-e1bc-4802-9e2a-69a5de755d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Jul 27 20:18:50.330013 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:50.329980    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:50.330490 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:50.330283    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:18:50.330966 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:18:50.330950    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:27:52.348680 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:52.348595    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7342-139753588705584"]
Jul 27 20:27:52.351946 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:52.351925    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-139753588705584"
Jul 27 20:27:52.354023 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:52.354004    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kljmn\""
Jul 27 20:27:52.358430 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:52.358409    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7342-139753588705584"]
Jul 27 20:27:52.406294 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:52.406261    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvg2t\" (UniqueName: \"kubernetes.io/projected/4a93a921-2ba2-4036-8970-34dea16cb1a3-kube-api-access-wvg2t\") pod \"test-curl-7342-139753588705584\" (UID: \"4a93a921-2ba2-4036-8970-34dea16cb1a3\") " pod="opendatahub/test-curl-7342-139753588705584"
Jul 27 20:27:52.507372 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:52.507337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvg2t\" (UniqueName: \"kubernetes.io/projected/4a93a921-2ba2-4036-8970-34dea16cb1a3-kube-api-access-wvg2t\") pod \"test-curl-7342-139753588705584\" (UID: \"4a93a921-2ba2-4036-8970-34dea16cb1a3\") " pod="opendatahub/test-curl-7342-139753588705584"
Jul 27 20:27:52.515095 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:52.515070    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvg2t\" (UniqueName: \"kubernetes.io/projected/4a93a921-2ba2-4036-8970-34dea16cb1a3-kube-api-access-wvg2t\") pod \"test-curl-7342-139753588705584\" (UID: \"4a93a921-2ba2-4036-8970-34dea16cb1a3\") " pod="opendatahub/test-curl-7342-139753588705584"
Jul 27 20:27:52.664110 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:52.664077    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-139753588705584"
Jul 27 20:27:52.807880 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:52.807846    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7342-139753588705584"]
Jul 27 20:27:52.809703 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:27:52.809676    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a93a921_2ba2_4036_8970_34dea16cb1a3.slice/crio-5c318efef6be0f5a806de5c888bcff969e8d69ad4560bb3cbfe34c825e6d30b5 WatchSource:0}: Error finding container 5c318efef6be0f5a806de5c888bcff969e8d69ad4560bb3cbfe34c825e6d30b5: Status 404 returned error can't find the container with id 5c318efef6be0f5a806de5c888bcff969e8d69ad4560bb3cbfe34c825e6d30b5
Jul 27 20:27:52.811006 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:52.810988    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:27:53.231841 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:53.231784    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7342-139753588705584" event={"ID":"4a93a921-2ba2-4036-8970-34dea16cb1a3","Type":"ContainerStarted","Data":"5c318efef6be0f5a806de5c888bcff969e8d69ad4560bb3cbfe34c825e6d30b5"}
Jul 27 20:27:54.239171 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:54.239135    2562 generic.go:356] "Generic (PLEG): container finished" podID="4a93a921-2ba2-4036-8970-34dea16cb1a3" containerID="5fde98c3f6ec103b29f707e74185664ddd46144f1f087d31a3dbdb1ff828ad0f" exitCode=0
Jul 27 20:27:54.239600 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:54.239224    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7342-139753588705584" event={"ID":"4a93a921-2ba2-4036-8970-34dea16cb1a3","Type":"ContainerDied","Data":"5fde98c3f6ec103b29f707e74185664ddd46144f1f087d31a3dbdb1ff828ad0f"}
Jul 27 20:27:55.388448 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:55.388416    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-139753588705584"
Jul 27 20:27:55.423888 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:55.423856    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7342-139753588705584_4a93a921-2ba2-4036-8970-34dea16cb1a3/test-curl-7342-139753588705584/0.log"
Jul 27 20:27:55.438215 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:55.438188    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvg2t\" (UniqueName: \"kubernetes.io/projected/4a93a921-2ba2-4036-8970-34dea16cb1a3-kube-api-access-wvg2t\") pod \"4a93a921-2ba2-4036-8970-34dea16cb1a3\" (UID: \"4a93a921-2ba2-4036-8970-34dea16cb1a3\") "
Jul 27 20:27:55.440472 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:55.440445    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a93a921-2ba2-4036-8970-34dea16cb1a3-kube-api-access-wvg2t" (OuterVolumeSpecName: "kube-api-access-wvg2t") pod "4a93a921-2ba2-4036-8970-34dea16cb1a3" (UID: "4a93a921-2ba2-4036-8970-34dea16cb1a3"). InnerVolumeSpecName "kube-api-access-wvg2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:27:55.447291 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:55.447269    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7342-139753588705584"]
Jul 27 20:27:55.451401 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:55.451381    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7342-139753588705584"]
Jul 27 20:27:55.539749 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:55.539663    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvg2t\" (UniqueName: \"kubernetes.io/projected/4a93a921-2ba2-4036-8970-34dea16cb1a3-kube-api-access-wvg2t\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:27:55.992053 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:55.992021    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a93a921-2ba2-4036-8970-34dea16cb1a3" path="/var/lib/kubelet/pods/4a93a921-2ba2-4036-8970-34dea16cb1a3/volumes"
Jul 27 20:27:56.251180 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:56.251085    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-139753588705584"
Jul 27 20:27:56.251180 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:56.251141    2562 scope.go:117] "RemoveContainer" containerID="5fde98c3f6ec103b29f707e74185664ddd46144f1f087d31a3dbdb1ff828ad0f"
Jul 27 20:27:58.348189 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:58.348152    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7342-139753583413120"]
Jul 27 20:27:58.348661 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:58.348645    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a93a921-2ba2-4036-8970-34dea16cb1a3" containerName="test-curl-7342-139753588705584"
Jul 27 20:27:58.348715 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:58.348663    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a93a921-2ba2-4036-8970-34dea16cb1a3" containerName="test-curl-7342-139753588705584"
Jul 27 20:27:58.348761 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:58.348727    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a93a921-2ba2-4036-8970-34dea16cb1a3" containerName="test-curl-7342-139753588705584"
Jul 27 20:27:58.356876 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:58.356849    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-139753583413120"
Jul 27 20:27:58.359935 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:58.359910    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kljmn\""
Jul 27 20:27:58.361808 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:58.361780    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7342-139753583413120"]
Jul 27 20:27:58.470403 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:58.470365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wrw2\" (UniqueName: \"kubernetes.io/projected/8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2-kube-api-access-9wrw2\") pod \"test-curl-7342-139753583413120\" (UID: \"8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2\") " pod="opendatahub/test-curl-7342-139753583413120"
Jul 27 20:27:58.570892 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:58.570849    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wrw2\" (UniqueName: \"kubernetes.io/projected/8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2-kube-api-access-9wrw2\") pod \"test-curl-7342-139753583413120\" (UID: \"8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2\") " pod="opendatahub/test-curl-7342-139753583413120"
Jul 27 20:27:58.579312 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:58.579278    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wrw2\" (UniqueName: \"kubernetes.io/projected/8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2-kube-api-access-9wrw2\") pod \"test-curl-7342-139753583413120\" (UID: \"8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2\") " pod="opendatahub/test-curl-7342-139753583413120"
Jul 27 20:27:58.670236 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:58.670205    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-139753583413120"
Jul 27 20:27:58.814232 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:58.814199    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7342-139753583413120"]
Jul 27 20:27:58.815831 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:27:58.815804    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8eb7d9f7_addc_4d93_99e6_9cce5a4e7af2.slice/crio-9d906ce65274b33d941b36dd3446052f46213bcfda5e06c97f4ca995d555afa5 WatchSource:0}: Error finding container 9d906ce65274b33d941b36dd3446052f46213bcfda5e06c97f4ca995d555afa5: Status 404 returned error can't find the container with id 9d906ce65274b33d941b36dd3446052f46213bcfda5e06c97f4ca995d555afa5
Jul 27 20:27:59.270491 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:59.270440    2562 generic.go:356] "Generic (PLEG): container finished" podID="8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2" containerID="67acbcc0206973f8bb6e5f0177985c8e290c44e8dda3e24cb6c0a2f5d484a907" exitCode=0
Jul 27 20:27:59.270665 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:59.270535    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7342-139753583413120" event={"ID":"8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2","Type":"ContainerDied","Data":"67acbcc0206973f8bb6e5f0177985c8e290c44e8dda3e24cb6c0a2f5d484a907"}
Jul 27 20:27:59.270665 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:27:59.270596    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7342-139753583413120" event={"ID":"8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2","Type":"ContainerStarted","Data":"9d906ce65274b33d941b36dd3446052f46213bcfda5e06c97f4ca995d555afa5"}
Jul 27 20:28:00.428826 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.428799    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-139753583413120"
Jul 27 20:28:00.447879 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.447852    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7342-139753583413120_8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2/test-curl-7342-139753583413120/0.log"
Jul 27 20:28:00.468013 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.467984    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7342-139753583413120"]
Jul 27 20:28:00.474855 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.474826    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7342-139753583413120"]
Jul 27 20:28:00.490506 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.490461    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wrw2\" (UniqueName: \"kubernetes.io/projected/8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2-kube-api-access-9wrw2\") pod \"8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2\" (UID: \"8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2\") "
Jul 27 20:28:00.492879 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.492852    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2-kube-api-access-9wrw2" (OuterVolumeSpecName: "kube-api-access-9wrw2") pod "8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2" (UID: "8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2"). InnerVolumeSpecName "kube-api-access-9wrw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:28:00.591112 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.591020    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7342-139753588852320"]
Jul 27 20:28:00.591611 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.591590    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2" containerName="test-curl-7342-139753583413120"
Jul 27 20:28:00.591713 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.591612    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2" containerName="test-curl-7342-139753583413120"
Jul 27 20:28:00.591713 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.591633    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wrw2\" (UniqueName: \"kubernetes.io/projected/8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2-kube-api-access-9wrw2\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:28:00.591823 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.591732    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2" containerName="test-curl-7342-139753583413120"
Jul 27 20:28:00.594920 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.594898    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-139753588852320"
Jul 27 20:28:00.599087 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.599055    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7342-139753588852320"]
Jul 27 20:28:00.692649 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.692613    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lsdg\" (UniqueName: \"kubernetes.io/projected/efce2a29-aa94-4d99-8ad3-ed9972f12098-kube-api-access-5lsdg\") pod \"test-curl-7342-139753588852320\" (UID: \"efce2a29-aa94-4d99-8ad3-ed9972f12098\") " pod="opendatahub/test-curl-7342-139753588852320"
Jul 27 20:28:00.793466 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.793428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lsdg\" (UniqueName: \"kubernetes.io/projected/efce2a29-aa94-4d99-8ad3-ed9972f12098-kube-api-access-5lsdg\") pod \"test-curl-7342-139753588852320\" (UID: \"efce2a29-aa94-4d99-8ad3-ed9972f12098\") " pod="opendatahub/test-curl-7342-139753588852320"
Jul 27 20:28:00.801390 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.801358    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lsdg\" (UniqueName: \"kubernetes.io/projected/efce2a29-aa94-4d99-8ad3-ed9972f12098-kube-api-access-5lsdg\") pod \"test-curl-7342-139753588852320\" (UID: \"efce2a29-aa94-4d99-8ad3-ed9972f12098\") " pod="opendatahub/test-curl-7342-139753588852320"
Jul 27 20:28:00.906656 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:00.906620    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-139753588852320"
Jul 27 20:28:01.050928 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:01.050891    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7342-139753588852320"]
Jul 27 20:28:01.052807 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:28:01.052782    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefce2a29_aa94_4d99_8ad3_ed9972f12098.slice/crio-0c7a39d9048b9527e61289ce3c971d32c0bc6c1ea179068190ee69c14eb39a8d WatchSource:0}: Error finding container 0c7a39d9048b9527e61289ce3c971d32c0bc6c1ea179068190ee69c14eb39a8d: Status 404 returned error can't find the container with id 0c7a39d9048b9527e61289ce3c971d32c0bc6c1ea179068190ee69c14eb39a8d
Jul 27 20:28:01.282913 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:01.282818    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7342-139753588852320" event={"ID":"efce2a29-aa94-4d99-8ad3-ed9972f12098","Type":"ContainerStarted","Data":"0c7a39d9048b9527e61289ce3c971d32c0bc6c1ea179068190ee69c14eb39a8d"}
Jul 27 20:28:01.284116 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:01.284091    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-139753583413120"
Jul 27 20:28:01.284116 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:01.284112    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d906ce65274b33d941b36dd3446052f46213bcfda5e06c97f4ca995d555afa5"
Jul 27 20:28:01.294449 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:28:01.294421    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7342-139753583413120\" is forbidden: User \"system:node:ip-10-0-142-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-140.ec2.internal' and this object" podUID="8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2" pod="opendatahub/test-curl-7342-139753583413120"
Jul 27 20:28:01.991681 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:01.991652    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2" path="/var/lib/kubelet/pods/8eb7d9f7-addc-4d93-99e6-9cce5a4e7af2/volumes"
Jul 27 20:28:02.292955 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:02.292916    2562 generic.go:356] "Generic (PLEG): container finished" podID="efce2a29-aa94-4d99-8ad3-ed9972f12098" containerID="51102d1f60e725206d775f69854ae8fc2823ad3ab42924d5f7856edd666783a1" exitCode=0
Jul 27 20:28:02.293391 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:02.293010    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7342-139753588852320" event={"ID":"efce2a29-aa94-4d99-8ad3-ed9972f12098","Type":"ContainerDied","Data":"51102d1f60e725206d775f69854ae8fc2823ad3ab42924d5f7856edd666783a1"}
Jul 27 20:28:03.439643 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:03.439617    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-139753588852320"
Jul 27 20:28:03.457343 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:03.457301    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7342-139753588852320_efce2a29-aa94-4d99-8ad3-ed9972f12098/test-curl-7342-139753588852320/0.log"
Jul 27 20:28:03.478861 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:03.478832    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7342-139753588852320"]
Jul 27 20:28:03.482407 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:03.482384    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7342-139753588852320"]
Jul 27 20:28:03.520895 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:03.520860    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lsdg\" (UniqueName: \"kubernetes.io/projected/efce2a29-aa94-4d99-8ad3-ed9972f12098-kube-api-access-5lsdg\") pod \"efce2a29-aa94-4d99-8ad3-ed9972f12098\" (UID: \"efce2a29-aa94-4d99-8ad3-ed9972f12098\") "
Jul 27 20:28:03.523098 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:03.523066    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efce2a29-aa94-4d99-8ad3-ed9972f12098-kube-api-access-5lsdg" (OuterVolumeSpecName: "kube-api-access-5lsdg") pod "efce2a29-aa94-4d99-8ad3-ed9972f12098" (UID: "efce2a29-aa94-4d99-8ad3-ed9972f12098"). InnerVolumeSpecName "kube-api-access-5lsdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:28:03.622585 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:03.622554    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lsdg\" (UniqueName: \"kubernetes.io/projected/efce2a29-aa94-4d99-8ad3-ed9972f12098-kube-api-access-5lsdg\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:28:03.992522 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:03.992422    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efce2a29-aa94-4d99-8ad3-ed9972f12098" path="/var/lib/kubelet/pods/efce2a29-aa94-4d99-8ad3-ed9972f12098/volumes"
Jul 27 20:28:04.305004 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:04.304919    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7342-139753588852320"
Jul 27 20:28:04.305004 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:04.304949    2562 scope.go:117] "RemoveContainer" containerID="51102d1f60e725206d775f69854ae8fc2823ad3ab42924d5f7856edd666783a1"
Jul 27 20:28:07.072989 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.072517    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"]
Jul 27 20:28:07.073504 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.073123    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efce2a29-aa94-4d99-8ad3-ed9972f12098" containerName="test-curl-7342-139753588852320"
Jul 27 20:28:07.073504 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.073140    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="efce2a29-aa94-4d99-8ad3-ed9972f12098" containerName="test-curl-7342-139753588852320"
Jul 27 20:28:07.073504 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.073224    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="efce2a29-aa94-4d99-8ad3-ed9972f12098" containerName="test-curl-7342-139753588852320"
Jul 27 20:28:07.077904 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.077881    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.080588 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.080566    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-555352-openshift-default-dockercfg-26xsd\""
Jul 27 20:28:07.090767 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.090738    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"]
Jul 27 20:28:07.158155 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.158116    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a671ce39-da00-4651-9314-777919d2a7f1-istio-podinfo\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.158425 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.158396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-workload-certs\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.158624 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.158462    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a671ce39-da00-4651-9314-777919d2a7f1-istiod-ca-cert\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.158624 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.158531    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a671ce39-da00-4651-9314-777919d2a7f1-istio-token\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.158624 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.158556    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djhmv\" (UniqueName: \"kubernetes.io/projected/a671ce39-da00-4651-9314-777919d2a7f1-kube-api-access-djhmv\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.158624 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.158580    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-credential-socket\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.158624 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.158613    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-istio-envoy\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.158936 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.158731    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-istio-data\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.158936 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.158770    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-workload-socket\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.260140 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.260101    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a671ce39-da00-4651-9314-777919d2a7f1-istio-podinfo\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.260349 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.260147    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-workload-certs\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.260349 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.260189    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a671ce39-da00-4651-9314-777919d2a7f1-istiod-ca-cert\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.260349 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.260223    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a671ce39-da00-4651-9314-777919d2a7f1-istio-token\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.260349 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.260246    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhmv\" (UniqueName: \"kubernetes.io/projected/a671ce39-da00-4651-9314-777919d2a7f1-kube-api-access-djhmv\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.260349 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.260267    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-credential-socket\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.260349 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.260295    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-istio-envoy\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.260717 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.260374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-istio-data\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.260717 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.260421    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-workload-socket\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.260717 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.260704    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-workload-certs\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.260882 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.260779    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-credential-socket\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.260882 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.260827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-workload-socket\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.261047 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.261030    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-istio-data\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.261205 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.261179    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a671ce39-da00-4651-9314-777919d2a7f1-istiod-ca-cert\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.262925 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.262898    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a671ce39-da00-4651-9314-777919d2a7f1-istio-podinfo\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.263032 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.263015    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-istio-envoy\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.267721 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.267701    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a671ce39-da00-4651-9314-777919d2a7f1-istio-token\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.267941 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.267923    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djhmv\" (UniqueName: \"kubernetes.io/projected/a671ce39-da00-4651-9314-777919d2a7f1-kube-api-access-djhmv\") pod \"e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") " pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.391460 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.391408    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:07.557965 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.557932    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"]
Jul 27 20:28:07.559419 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:28:07.559392    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda671ce39_da00_4651_9314_777919d2a7f1.slice/crio-0b7eb442f67ef208177599f9e2c7c62cc047368a2588b7226a3ff7f04732b465 WatchSource:0}: Error finding container 0b7eb442f67ef208177599f9e2c7c62cc047368a2588b7226a3ff7f04732b465: Status 404 returned error can't find the container with id 0b7eb442f67ef208177599f9e2c7c62cc047368a2588b7226a3ff7f04732b465
Jul 27 20:28:07.561787 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.561749    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:28:07.561895 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.561817    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:28:07.561895 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:07.561844    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:28:08.328837 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:08.328793    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx" event={"ID":"a671ce39-da00-4651-9314-777919d2a7f1","Type":"ContainerStarted","Data":"3f0b72367d407f0525085dcdad4c805480e7236a67e4d757c526ee296e7a803b"}
Jul 27 20:28:08.329201 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:08.328867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx" event={"ID":"a671ce39-da00-4651-9314-777919d2a7f1","Type":"ContainerStarted","Data":"0b7eb442f67ef208177599f9e2c7c62cc047368a2588b7226a3ff7f04732b465"}
Jul 27 20:28:08.347234 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:08.347178    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx" podStartSLOduration=1.347161964 podStartE2EDuration="1.347161964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:28:08.345593828 +0000 UTC m=+2902.897510246" watchObservedRunningTime="2026-07-27 20:28:08.347161964 +0000 UTC m=+2902.899078382"
Jul 27 20:28:08.391877 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:08.391841    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:08.397342 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:08.397318    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:09.335800 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:09.335768    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:09.336965 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:09.336942    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:11.655723 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:11.655680    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"]
Jul 27 20:28:11.660417 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:11.660388    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:28:11.662499 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:11.662459    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-555352-serving-cert\""
Jul 27 20:28:11.673324 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:11.673280    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"]
Jul 27 20:28:11.810638 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:11.810597    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bdbbac-3b26-4549-a4b5-82a997f3f579-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq\" (UID: \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:28:11.810818 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:11.810646    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8qvv\" (UniqueName: \"kubernetes.io/projected/c3bdbbac-3b26-4549-a4b5-82a997f3f579-kube-api-access-t8qvv\") pod \"maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq\" (UID: \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:28:11.810818 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:11.810734    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3bdbbac-3b26-4549-a4b5-82a997f3f579-maas-api-tls\") pod \"maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq\" (UID: \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:28:11.911930 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:11.911844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3bdbbac-3b26-4549-a4b5-82a997f3f579-maas-api-tls\") pod \"maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq\" (UID: \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:28:11.912083 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:11.911936    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bdbbac-3b26-4549-a4b5-82a997f3f579-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq\" (UID: \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:28:11.912083 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:11.911966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8qvv\" (UniqueName: \"kubernetes.io/projected/c3bdbbac-3b26-4549-a4b5-82a997f3f579-kube-api-access-t8qvv\") pod \"maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq\" (UID: \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:28:11.912690 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:11.912668    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bdbbac-3b26-4549-a4b5-82a997f3f579-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq\" (UID: \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:28:11.914459 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:11.914436    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3bdbbac-3b26-4549-a4b5-82a997f3f579-maas-api-tls\") pod \"maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq\" (UID: \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:28:11.919159 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:11.919133    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8qvv\" (UniqueName: \"kubernetes.io/projected/c3bdbbac-3b26-4549-a4b5-82a997f3f579-kube-api-access-t8qvv\") pod \"maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq\" (UID: \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:28:11.974347 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:11.974315    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:28:12.131303 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:12.128977    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"]
Jul 27 20:28:12.131303 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:28:12.129850    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3bdbbac_3b26_4549_a4b5_82a997f3f579.slice/crio-1fb2ea792fb0f7dd9a8cec209f36d6427b7d0089b39dccb77f7104f0b4b81bc7 WatchSource:0}: Error finding container 1fb2ea792fb0f7dd9a8cec209f36d6427b7d0089b39dccb77f7104f0b4b81bc7: Status 404 returned error can't find the container with id 1fb2ea792fb0f7dd9a8cec209f36d6427b7d0089b39dccb77f7104f0b4b81bc7
Jul 27 20:28:12.353514 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:12.353402    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq" event={"ID":"c3bdbbac-3b26-4549-a4b5-82a997f3f579","Type":"ContainerStarted","Data":"1fb2ea792fb0f7dd9a8cec209f36d6427b7d0089b39dccb77f7104f0b4b81bc7"}
Jul 27 20:28:13.361158 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:13.361127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq" event={"ID":"c3bdbbac-3b26-4549-a4b5-82a997f3f579","Type":"ContainerStarted","Data":"58e78f2809ee2a22bdd8fb328a1b2f1c9eee5325ffb06c2e896150a645b745b4"}
Jul 27 20:28:13.361560 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:13.361197    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:28:13.383033 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:13.382972    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq" podStartSLOduration=1.853238217 podStartE2EDuration="2.382951219s" totalImagesPullingTime="529.713002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:28:13.379644623 +0000 UTC m=+2907.931561042" watchObservedRunningTime="2026-07-27 20:28:13.382951219 +0000 UTC m=+2907.934867640"
Jul 27 20:28:19.373010 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:19.372978    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:28:28.456807 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:28.456772    2562 generic.go:356] "Generic (PLEG): container finished" podID="a671ce39-da00-4651-9314-777919d2a7f1" containerID="3f0b72367d407f0525085dcdad4c805480e7236a67e4d757c526ee296e7a803b" exitCode=0
Jul 27 20:28:28.457238 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:28.456843    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx" event={"ID":"a671ce39-da00-4651-9314-777919d2a7f1","Type":"ContainerDied","Data":"3f0b72367d407f0525085dcdad4c805480e7236a67e4d757c526ee296e7a803b"}
Jul 27 20:28:28.457285 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:28.457266    2562 scope.go:117] "RemoveContainer" containerID="3f0b72367d407f0525085dcdad4c805480e7236a67e4d757c526ee296e7a803b"
Jul 27 20:28:28.458406 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:28.458381    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:28:28.458489 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:28.458446    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:28:28.458535 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:28.458498    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 27 20:28:29.464920 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:29.464883    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx" event={"ID":"a671ce39-da00-4651-9314-777919d2a7f1","Type":"ContainerStarted","Data":"9942fee6d3b840828df4263814e3543d1e7dd39e83b690bc5df189aea3b79b2e"}
Jul 27 20:28:30.392323 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:30.392287    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:30.392627 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:30.392591    2562 patch_prober.go:28] interesting pod/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Jul 27 20:28:30.392719 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:30.392659    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx" podUID="a671ce39-da00-4651-9314-777919d2a7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Jul 27 20:28:31.392465 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:31.392424    2562 patch_prober.go:28] interesting pod/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Jul 27 20:28:31.393095 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:31.393061    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx" podUID="a671ce39-da00-4651-9314-777919d2a7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Jul 27 20:28:32.392453 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:32.392414    2562 patch_prober.go:28] interesting pod/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Jul 27 20:28:32.392766 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:32.392736    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx" podUID="a671ce39-da00-4651-9314-777919d2a7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Jul 27 20:28:33.395202 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:33.395169    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:33.395607 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:33.395588    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:33.396344 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:33.396329    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:28:33.809848 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:33.809740    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"]
Jul 27 20:28:33.812880 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:33.812857    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:28:33.814878 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:33.814855    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1517d6-serving-cert\""
Jul 27 20:28:33.824341 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:33.824318    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"]
Jul 27 20:28:33.947840 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:33.947803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcj2d\" (UniqueName: \"kubernetes.io/projected/822479a1-9e4f-4bbf-b712-ee4985917b64-kube-api-access-mcj2d\") pod \"maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m\" (UID: \"822479a1-9e4f-4bbf-b712-ee4985917b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:28:33.948008 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:33.947853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/822479a1-9e4f-4bbf-b712-ee4985917b64-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m\" (UID: \"822479a1-9e4f-4bbf-b712-ee4985917b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:28:33.948008 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:33.947910    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/822479a1-9e4f-4bbf-b712-ee4985917b64-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m\" (UID: \"822479a1-9e4f-4bbf-b712-ee4985917b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:28:34.049107 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:34.049069    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcj2d\" (UniqueName: \"kubernetes.io/projected/822479a1-9e4f-4bbf-b712-ee4985917b64-kube-api-access-mcj2d\") pod \"maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m\" (UID: \"822479a1-9e4f-4bbf-b712-ee4985917b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:28:34.049397 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:34.049374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/822479a1-9e4f-4bbf-b712-ee4985917b64-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m\" (UID: \"822479a1-9e4f-4bbf-b712-ee4985917b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:28:34.049519 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:34.049428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/822479a1-9e4f-4bbf-b712-ee4985917b64-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m\" (UID: \"822479a1-9e4f-4bbf-b712-ee4985917b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:28:34.049587 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:28:34.049558    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-serving-cert: secret "maas-api-e2e-ipp-b-1517d6-serving-cert" not found
Jul 27 20:28:34.049644 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:28:34.049619    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/822479a1-9e4f-4bbf-b712-ee4985917b64-maas-api-tls podName:822479a1-9e4f-4bbf-b712-ee4985917b64 nodeName:}" failed. No retries permitted until 2026-07-27 20:28:34.54959773 +0000 UTC m=+2929.101514127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/822479a1-9e4f-4bbf-b712-ee4985917b64-maas-api-tls") pod "maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m" (UID: "822479a1-9e4f-4bbf-b712-ee4985917b64") : secret "maas-api-e2e-ipp-b-1517d6-serving-cert" not found
Jul 27 20:28:34.050052 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:34.050031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/822479a1-9e4f-4bbf-b712-ee4985917b64-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m\" (UID: \"822479a1-9e4f-4bbf-b712-ee4985917b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:28:34.057446 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:34.057426    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcj2d\" (UniqueName: \"kubernetes.io/projected/822479a1-9e4f-4bbf-b712-ee4985917b64-kube-api-access-mcj2d\") pod \"maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m\" (UID: \"822479a1-9e4f-4bbf-b712-ee4985917b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:28:34.555962 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:34.555914    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/822479a1-9e4f-4bbf-b712-ee4985917b64-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m\" (UID: \"822479a1-9e4f-4bbf-b712-ee4985917b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:28:34.558737 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:34.558707    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/822479a1-9e4f-4bbf-b712-ee4985917b64-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m\" (UID: \"822479a1-9e4f-4bbf-b712-ee4985917b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:28:34.725984 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:34.725945    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:28:35.082250 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:35.082216    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"]
Jul 27 20:28:35.084499 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:28:35.084437    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822479a1_9e4f_4bbf_b712_ee4985917b64.slice/crio-933fbcaa6d8e40e99730f3665dad54111c0d93d45221be67c88d76155a7a5216 WatchSource:0}: Error finding container 933fbcaa6d8e40e99730f3665dad54111c0d93d45221be67c88d76155a7a5216: Status 404 returned error can't find the container with id 933fbcaa6d8e40e99730f3665dad54111c0d93d45221be67c88d76155a7a5216
Jul 27 20:28:35.504673 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:35.504640    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m" event={"ID":"822479a1-9e4f-4bbf-b712-ee4985917b64","Type":"ContainerStarted","Data":"933fbcaa6d8e40e99730f3665dad54111c0d93d45221be67c88d76155a7a5216"}
Jul 27 20:28:36.511851 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:36.511804    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m" event={"ID":"822479a1-9e4f-4bbf-b712-ee4985917b64","Type":"ContainerStarted","Data":"1c5a62c2d0d1bc2a6771f009154730ae33dbc8c4e4152186a8f2eaff2322e638"}
Jul 27 20:28:36.512282 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:36.511939    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:28:36.529040 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:36.528987    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m" podStartSLOduration=3.110019789 podStartE2EDuration="3.528970364s" totalImagesPullingTime="418.950575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:28:36.526584471 +0000 UTC m=+2931.078500890" watchObservedRunningTime="2026-07-27 20:28:36.528970364 +0000 UTC m=+2931.080886807"
Jul 27 20:28:42.523712 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:42.523678    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:28:53.619318 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:28:53.619284    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wrscr_65bda14a-6ab1-406e-8754-305218a633ab/payload-processing/0.log"
Jul 27 20:29:53.991589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:29:53.991563    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wrscr_65bda14a-6ab1-406e-8754-305218a633ab/payload-processing/0.log"
Jul 27 20:30:00.229957 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:00.229909    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"]
Jul 27 20:30:00.234597 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:00.234571    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:00.236609 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:00.236582    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-63bd67-serving-cert\""
Jul 27 20:30:00.241204 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:00.241178    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"]
Jul 27 20:30:00.333886 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:00.333851    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2rn\" (UniqueName: \"kubernetes.io/projected/c0aab554-59e6-4908-8d29-18ba81008ebc-kube-api-access-zm2rn\") pod \"maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn\" (UID: \"c0aab554-59e6-4908-8d29-18ba81008ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:00.334071 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:00.333896    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0aab554-59e6-4908-8d29-18ba81008ebc-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn\" (UID: \"c0aab554-59e6-4908-8d29-18ba81008ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:00.334071 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:00.334039    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0aab554-59e6-4908-8d29-18ba81008ebc-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn\" (UID: \"c0aab554-59e6-4908-8d29-18ba81008ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:00.434891 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:00.434861    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0aab554-59e6-4908-8d29-18ba81008ebc-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn\" (UID: \"c0aab554-59e6-4908-8d29-18ba81008ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:00.435051 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:00.434928    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2rn\" (UniqueName: \"kubernetes.io/projected/c0aab554-59e6-4908-8d29-18ba81008ebc-kube-api-access-zm2rn\") pod \"maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn\" (UID: \"c0aab554-59e6-4908-8d29-18ba81008ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:00.435051 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:00.434963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0aab554-59e6-4908-8d29-18ba81008ebc-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn\" (UID: \"c0aab554-59e6-4908-8d29-18ba81008ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:00.435163 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:30:00.435140    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-serving-cert: secret "maas-api-e2e-ipp-cleanup-63bd67-serving-cert" not found
Jul 27 20:30:00.435251 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:30:00.435239    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0aab554-59e6-4908-8d29-18ba81008ebc-maas-api-tls podName:c0aab554-59e6-4908-8d29-18ba81008ebc nodeName:}" failed. No retries permitted until 2026-07-27 20:30:00.935216663 +0000 UTC m=+3015.487133077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0aab554-59e6-4908-8d29-18ba81008ebc-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn" (UID: "c0aab554-59e6-4908-8d29-18ba81008ebc") : secret "maas-api-e2e-ipp-cleanup-63bd67-serving-cert" not found
Jul 27 20:30:00.435523 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:00.435506    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0aab554-59e6-4908-8d29-18ba81008ebc-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn\" (UID: \"c0aab554-59e6-4908-8d29-18ba81008ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:00.442955 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:00.442929    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2rn\" (UniqueName: \"kubernetes.io/projected/c0aab554-59e6-4908-8d29-18ba81008ebc-kube-api-access-zm2rn\") pod \"maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn\" (UID: \"c0aab554-59e6-4908-8d29-18ba81008ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:00.939778 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:00.939737    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0aab554-59e6-4908-8d29-18ba81008ebc-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn\" (UID: \"c0aab554-59e6-4908-8d29-18ba81008ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:00.942265 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:00.942236    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0aab554-59e6-4908-8d29-18ba81008ebc-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn\" (UID: \"c0aab554-59e6-4908-8d29-18ba81008ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:01.149709 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:01.149673    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:01.298210 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:01.298183    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"]
Jul 27 20:30:01.300987 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:30:01.300958    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0aab554_59e6_4908_8d29_18ba81008ebc.slice/crio-9a0933e975330353d2734fd6f5bc7ff4c43eb81749ed8bcc45a34ab658c5f881 WatchSource:0}: Error finding container 9a0933e975330353d2734fd6f5bc7ff4c43eb81749ed8bcc45a34ab658c5f881: Status 404 returned error can't find the container with id 9a0933e975330353d2734fd6f5bc7ff4c43eb81749ed8bcc45a34ab658c5f881
Jul 27 20:30:02.002105 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:02.002068    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn" event={"ID":"c0aab554-59e6-4908-8d29-18ba81008ebc","Type":"ContainerStarted","Data":"9fed3cf7b1ca72031a6e14a8a1dcc91417be3bc02f6ce77a0973985ff88b130b"}
Jul 27 20:30:02.002262 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:02.002126    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn" event={"ID":"c0aab554-59e6-4908-8d29-18ba81008ebc","Type":"ContainerStarted","Data":"9a0933e975330353d2734fd6f5bc7ff4c43eb81749ed8bcc45a34ab658c5f881"}
Jul 27 20:30:02.002262 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:02.002250    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:02.076176 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:02.076123    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn" podStartSLOduration=1.478705785 podStartE2EDuration="2.076104961s" totalImagesPullingTime="597.399176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:30:02.072187707 +0000 UTC m=+3016.624104126" watchObservedRunningTime="2026-07-27 20:30:02.076104961 +0000 UTC m=+3016.628021410"
Jul 27 20:30:08.014281 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:08.014226    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:28.879813 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:28.879771    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"]
Jul 27 20:30:28.880289 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:28.880032    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn" podUID="c0aab554-59e6-4908-8d29-18ba81008ebc" containerName="maas-api" containerID="cri-o://9fed3cf7b1ca72031a6e14a8a1dcc91417be3bc02f6ce77a0973985ff88b130b" gracePeriod=30
Jul 27 20:30:29.159685 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:29.159655    2562 generic.go:356] "Generic (PLEG): container finished" podID="c0aab554-59e6-4908-8d29-18ba81008ebc" containerID="9fed3cf7b1ca72031a6e14a8a1dcc91417be3bc02f6ce77a0973985ff88b130b" exitCode=0
Jul 27 20:30:29.159809 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:29.159733    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn" event={"ID":"c0aab554-59e6-4908-8d29-18ba81008ebc","Type":"ContainerDied","Data":"9fed3cf7b1ca72031a6e14a8a1dcc91417be3bc02f6ce77a0973985ff88b130b"}
Jul 27 20:30:29.159809 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:29.159791    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn" event={"ID":"c0aab554-59e6-4908-8d29-18ba81008ebc","Type":"ContainerDied","Data":"9a0933e975330353d2734fd6f5bc7ff4c43eb81749ed8bcc45a34ab658c5f881"}
Jul 27 20:30:29.159887 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:29.159808    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a0933e975330353d2734fd6f5bc7ff4c43eb81749ed8bcc45a34ab658c5f881"
Jul 27 20:30:29.167142 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:29.167119    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:29.337335 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:29.337299    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm2rn\" (UniqueName: \"kubernetes.io/projected/c0aab554-59e6-4908-8d29-18ba81008ebc-kube-api-access-zm2rn\") pod \"c0aab554-59e6-4908-8d29-18ba81008ebc\" (UID: \"c0aab554-59e6-4908-8d29-18ba81008ebc\") "
Jul 27 20:30:29.337563 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:29.337340    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0aab554-59e6-4908-8d29-18ba81008ebc-openshift-service-ca\") pod \"c0aab554-59e6-4908-8d29-18ba81008ebc\" (UID: \"c0aab554-59e6-4908-8d29-18ba81008ebc\") "
Jul 27 20:30:29.337563 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:29.337403    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0aab554-59e6-4908-8d29-18ba81008ebc-maas-api-tls\") pod \"c0aab554-59e6-4908-8d29-18ba81008ebc\" (UID: \"c0aab554-59e6-4908-8d29-18ba81008ebc\") "
Jul 27 20:30:29.337735 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:29.337709    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0aab554-59e6-4908-8d29-18ba81008ebc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c0aab554-59e6-4908-8d29-18ba81008ebc" (UID: "c0aab554-59e6-4908-8d29-18ba81008ebc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:30:29.337812 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:29.337798    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0aab554-59e6-4908-8d29-18ba81008ebc-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:30:29.339753 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:29.339725    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0aab554-59e6-4908-8d29-18ba81008ebc-kube-api-access-zm2rn" (OuterVolumeSpecName: "kube-api-access-zm2rn") pod "c0aab554-59e6-4908-8d29-18ba81008ebc" (UID: "c0aab554-59e6-4908-8d29-18ba81008ebc"). InnerVolumeSpecName "kube-api-access-zm2rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:30:29.339862 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:29.339764    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0aab554-59e6-4908-8d29-18ba81008ebc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c0aab554-59e6-4908-8d29-18ba81008ebc" (UID: "c0aab554-59e6-4908-8d29-18ba81008ebc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:30:29.439128 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:29.439028    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0aab554-59e6-4908-8d29-18ba81008ebc-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:30:29.439128 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:29.439058    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm2rn\" (UniqueName: \"kubernetes.io/projected/c0aab554-59e6-4908-8d29-18ba81008ebc-kube-api-access-zm2rn\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:30:30.164956 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:30.164922    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"
Jul 27 20:30:30.181961 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:30.181930    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"]
Jul 27 20:30:30.184242 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:30.184216    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-63bd67-6c7d7cb9d9-czfpn"]
Jul 27 20:30:31.992237 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:30:31.992208    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0aab554-59e6-4908-8d29-18ba81008ebc" path="/var/lib/kubelet/pods/c0aab554-59e6-4908-8d29-18ba81008ebc/volumes"
Jul 27 20:31:04.284388 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:04.283557    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"]
Jul 27 20:31:04.284989 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:04.284465    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq" podUID="c3bdbbac-3b26-4549-a4b5-82a997f3f579" containerName="maas-api" containerID="cri-o://58e78f2809ee2a22bdd8fb328a1b2f1c9eee5325ffb06c2e896150a645b745b4" gracePeriod=30
Jul 27 20:31:04.367368 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:04.367320    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq" podUID="c3bdbbac-3b26-4549-a4b5-82a997f3f579" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.80:8443/health\": dial tcp 10.133.0.80:8443: connect: connection refused"
Jul 27 20:31:04.568491 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:04.568454    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:31:04.613352 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:04.613318    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8qvv\" (UniqueName: \"kubernetes.io/projected/c3bdbbac-3b26-4549-a4b5-82a997f3f579-kube-api-access-t8qvv\") pod \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\" (UID: \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\") "
Jul 27 20:31:04.613552 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:04.613427    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3bdbbac-3b26-4549-a4b5-82a997f3f579-maas-api-tls\") pod \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\" (UID: \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\") "
Jul 27 20:31:04.613552 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:04.613508    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bdbbac-3b26-4549-a4b5-82a997f3f579-openshift-service-ca\") pod \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\" (UID: \"c3bdbbac-3b26-4549-a4b5-82a997f3f579\") "
Jul 27 20:31:04.613856 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:04.613834    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3bdbbac-3b26-4549-a4b5-82a997f3f579-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3bdbbac-3b26-4549-a4b5-82a997f3f579" (UID: "c3bdbbac-3b26-4549-a4b5-82a997f3f579"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:31:04.615580 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:04.615549    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3bdbbac-3b26-4549-a4b5-82a997f3f579-kube-api-access-t8qvv" (OuterVolumeSpecName: "kube-api-access-t8qvv") pod "c3bdbbac-3b26-4549-a4b5-82a997f3f579" (UID: "c3bdbbac-3b26-4549-a4b5-82a997f3f579"). InnerVolumeSpecName "kube-api-access-t8qvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:31:04.615725 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:04.615703    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3bdbbac-3b26-4549-a4b5-82a997f3f579-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3bdbbac-3b26-4549-a4b5-82a997f3f579" (UID: "c3bdbbac-3b26-4549-a4b5-82a997f3f579"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:31:04.714443 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:04.714405    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bdbbac-3b26-4549-a4b5-82a997f3f579-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:31:04.714443 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:04.714435    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8qvv\" (UniqueName: \"kubernetes.io/projected/c3bdbbac-3b26-4549-a4b5-82a997f3f579-kube-api-access-t8qvv\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:31:04.714443 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:04.714445    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3bdbbac-3b26-4549-a4b5-82a997f3f579-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:31:05.366642 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:05.366606    2562 generic.go:356] "Generic (PLEG): container finished" podID="c3bdbbac-3b26-4549-a4b5-82a997f3f579" containerID="58e78f2809ee2a22bdd8fb328a1b2f1c9eee5325ffb06c2e896150a645b745b4" exitCode=0
Jul 27 20:31:05.366642 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:05.366643    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"
Jul 27 20:31:05.367084 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:05.366688    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq" event={"ID":"c3bdbbac-3b26-4549-a4b5-82a997f3f579","Type":"ContainerDied","Data":"58e78f2809ee2a22bdd8fb328a1b2f1c9eee5325ffb06c2e896150a645b745b4"}
Jul 27 20:31:05.367084 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:05.366742    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq" event={"ID":"c3bdbbac-3b26-4549-a4b5-82a997f3f579","Type":"ContainerDied","Data":"1fb2ea792fb0f7dd9a8cec209f36d6427b7d0089b39dccb77f7104f0b4b81bc7"}
Jul 27 20:31:05.367084 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:05.366766    2562 scope.go:117] "RemoveContainer" containerID="58e78f2809ee2a22bdd8fb328a1b2f1c9eee5325ffb06c2e896150a645b745b4"
Jul 27 20:31:05.381754 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:05.381729    2562 scope.go:117] "RemoveContainer" containerID="58e78f2809ee2a22bdd8fb328a1b2f1c9eee5325ffb06c2e896150a645b745b4"
Jul 27 20:31:05.382608 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:31:05.382579    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e78f2809ee2a22bdd8fb328a1b2f1c9eee5325ffb06c2e896150a645b745b4\": container with ID starting with 58e78f2809ee2a22bdd8fb328a1b2f1c9eee5325ffb06c2e896150a645b745b4 not found: ID does not exist" containerID="58e78f2809ee2a22bdd8fb328a1b2f1c9eee5325ffb06c2e896150a645b745b4"
Jul 27 20:31:05.382720 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:05.382616    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e78f2809ee2a22bdd8fb328a1b2f1c9eee5325ffb06c2e896150a645b745b4"} err="failed to get container status \"58e78f2809ee2a22bdd8fb328a1b2f1c9eee5325ffb06c2e896150a645b745b4\": rpc error: code = NotFound desc = could not find container \"58e78f2809ee2a22bdd8fb328a1b2f1c9eee5325ffb06c2e896150a645b745b4\": container with ID starting with 58e78f2809ee2a22bdd8fb328a1b2f1c9eee5325ffb06c2e896150a645b745b4 not found: ID does not exist"
Jul 27 20:31:05.391964 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:05.391942    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"]
Jul 27 20:31:05.397005 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:05.396983    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-555352-85dd4c8f55-t55wq"]
Jul 27 20:31:05.991527 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:31:05.991473    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3bdbbac-3b26-4549-a4b5-82a997f3f579" path="/var/lib/kubelet/pods/c3bdbbac-3b26-4549-a4b5-82a997f3f579/volumes"
Jul 27 20:32:12.790013 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:12.789970    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"]
Jul 27 20:32:12.790591 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:12.790286    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx" podUID="a671ce39-da00-4651-9314-777919d2a7f1" containerName="istio-proxy" containerID="cri-o://9942fee6d3b840828df4263814e3543d1e7dd39e83b690bc5df189aea3b79b2e" gracePeriod=30
Jul 27 20:32:18.095472 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.095444    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:32:18.252961 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.252929    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-workload-socket\") pod \"a671ce39-da00-4651-9314-777919d2a7f1\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") "
Jul 27 20:32:18.252961 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.252961    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a671ce39-da00-4651-9314-777919d2a7f1-istio-podinfo\") pod \"a671ce39-da00-4651-9314-777919d2a7f1\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") "
Jul 27 20:32:18.253219 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.252983    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a671ce39-da00-4651-9314-777919d2a7f1-istio-token\") pod \"a671ce39-da00-4651-9314-777919d2a7f1\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") "
Jul 27 20:32:18.253219 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.253000    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-istio-envoy\") pod \"a671ce39-da00-4651-9314-777919d2a7f1\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") "
Jul 27 20:32:18.253219 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.253022    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-istio-data\") pod \"a671ce39-da00-4651-9314-777919d2a7f1\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") "
Jul 27 20:32:18.253219 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.253056    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-workload-certs\") pod \"a671ce39-da00-4651-9314-777919d2a7f1\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") "
Jul 27 20:32:18.253219 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.253142    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-credential-socket\") pod \"a671ce39-da00-4651-9314-777919d2a7f1\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") "
Jul 27 20:32:18.253219 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.253201    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djhmv\" (UniqueName: \"kubernetes.io/projected/a671ce39-da00-4651-9314-777919d2a7f1-kube-api-access-djhmv\") pod \"a671ce39-da00-4651-9314-777919d2a7f1\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") "
Jul 27 20:32:18.253569 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.253241    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a671ce39-da00-4651-9314-777919d2a7f1-istiod-ca-cert\") pod \"a671ce39-da00-4651-9314-777919d2a7f1\" (UID: \"a671ce39-da00-4651-9314-777919d2a7f1\") "
Jul 27 20:32:18.253569 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.253259    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a671ce39-da00-4651-9314-777919d2a7f1" (UID: "a671ce39-da00-4651-9314-777919d2a7f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:32:18.253569 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.253407    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a671ce39-da00-4651-9314-777919d2a7f1" (UID: "a671ce39-da00-4651-9314-777919d2a7f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:32:18.253727 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.253600    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a671ce39-da00-4651-9314-777919d2a7f1" (UID: "a671ce39-da00-4651-9314-777919d2a7f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:32:18.253727 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.253646    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-workload-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:18.253727 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.253685    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-credential-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:18.254060 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.253738    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a671ce39-da00-4651-9314-777919d2a7f1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a671ce39-da00-4651-9314-777919d2a7f1" (UID: "a671ce39-da00-4651-9314-777919d2a7f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:32:18.254060 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.254013    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-istio-data" (OuterVolumeSpecName: "istio-data") pod "a671ce39-da00-4651-9314-777919d2a7f1" (UID: "a671ce39-da00-4651-9314-777919d2a7f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:32:18.255382 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.255357    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a671ce39-da00-4651-9314-777919d2a7f1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a671ce39-da00-4651-9314-777919d2a7f1" (UID: "a671ce39-da00-4651-9314-777919d2a7f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:32:18.255627 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.255603    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a671ce39-da00-4651-9314-777919d2a7f1-kube-api-access-djhmv" (OuterVolumeSpecName: "kube-api-access-djhmv") pod "a671ce39-da00-4651-9314-777919d2a7f1" (UID: "a671ce39-da00-4651-9314-777919d2a7f1"). InnerVolumeSpecName "kube-api-access-djhmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:32:18.255834 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.255814    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a671ce39-da00-4651-9314-777919d2a7f1" (UID: "a671ce39-da00-4651-9314-777919d2a7f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:32:18.255924 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.255862    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a671ce39-da00-4651-9314-777919d2a7f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "a671ce39-da00-4651-9314-777919d2a7f1" (UID: "a671ce39-da00-4651-9314-777919d2a7f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:32:18.354763 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.354679    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a671ce39-da00-4651-9314-777919d2a7f1-istio-podinfo\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:18.354763 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.354706    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a671ce39-da00-4651-9314-777919d2a7f1-istio-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:18.354763 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.354717    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-istio-envoy\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:18.354763 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.354726    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-istio-data\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:18.354763 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.354734    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a671ce39-da00-4651-9314-777919d2a7f1-workload-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:18.354763 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.354742    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djhmv\" (UniqueName: \"kubernetes.io/projected/a671ce39-da00-4651-9314-777919d2a7f1-kube-api-access-djhmv\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:18.354763 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.354751    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a671ce39-da00-4651-9314-777919d2a7f1-istiod-ca-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:18.775957 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.775919    2562 generic.go:356] "Generic (PLEG): container finished" podID="a671ce39-da00-4651-9314-777919d2a7f1" containerID="9942fee6d3b840828df4263814e3543d1e7dd39e83b690bc5df189aea3b79b2e" exitCode=0
Jul 27 20:32:18.776124 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.775974    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"
Jul 27 20:32:18.776124 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.776001    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx" event={"ID":"a671ce39-da00-4651-9314-777919d2a7f1","Type":"ContainerDied","Data":"9942fee6d3b840828df4263814e3543d1e7dd39e83b690bc5df189aea3b79b2e"}
Jul 27 20:32:18.776124 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.776065    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx" event={"ID":"a671ce39-da00-4651-9314-777919d2a7f1","Type":"ContainerDied","Data":"0b7eb442f67ef208177599f9e2c7c62cc047368a2588b7226a3ff7f04732b465"}
Jul 27 20:32:18.776124 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.776089    2562 scope.go:117] "RemoveContainer" containerID="9942fee6d3b840828df4263814e3543d1e7dd39e83b690bc5df189aea3b79b2e"
Jul 27 20:32:18.786292 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.786274    2562 scope.go:117] "RemoveContainer" containerID="3f0b72367d407f0525085dcdad4c805480e7236a67e4d757c526ee296e7a803b"
Jul 27 20:32:18.796041 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.796022    2562 scope.go:117] "RemoveContainer" containerID="9942fee6d3b840828df4263814e3543d1e7dd39e83b690bc5df189aea3b79b2e"
Jul 27 20:32:18.796338 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:32:18.796322    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9942fee6d3b840828df4263814e3543d1e7dd39e83b690bc5df189aea3b79b2e\": container with ID starting with 9942fee6d3b840828df4263814e3543d1e7dd39e83b690bc5df189aea3b79b2e not found: ID does not exist" containerID="9942fee6d3b840828df4263814e3543d1e7dd39e83b690bc5df189aea3b79b2e"
Jul 27 20:32:18.796425 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.796346    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9942fee6d3b840828df4263814e3543d1e7dd39e83b690bc5df189aea3b79b2e"} err="failed to get container status \"9942fee6d3b840828df4263814e3543d1e7dd39e83b690bc5df189aea3b79b2e\": rpc error: code = NotFound desc = could not find container \"9942fee6d3b840828df4263814e3543d1e7dd39e83b690bc5df189aea3b79b2e\": container with ID starting with 9942fee6d3b840828df4263814e3543d1e7dd39e83b690bc5df189aea3b79b2e not found: ID does not exist"
Jul 27 20:32:18.796425 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.796362    2562 scope.go:117] "RemoveContainer" containerID="3f0b72367d407f0525085dcdad4c805480e7236a67e4d757c526ee296e7a803b"
Jul 27 20:32:18.796670 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:32:18.796647    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0b72367d407f0525085dcdad4c805480e7236a67e4d757c526ee296e7a803b\": container with ID starting with 3f0b72367d407f0525085dcdad4c805480e7236a67e4d757c526ee296e7a803b not found: ID does not exist" containerID="3f0b72367d407f0525085dcdad4c805480e7236a67e4d757c526ee296e7a803b"
Jul 27 20:32:18.796709 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.796681    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0b72367d407f0525085dcdad4c805480e7236a67e4d757c526ee296e7a803b"} err="failed to get container status \"3f0b72367d407f0525085dcdad4c805480e7236a67e4d757c526ee296e7a803b\": rpc error: code = NotFound desc = could not find container \"3f0b72367d407f0525085dcdad4c805480e7236a67e4d757c526ee296e7a803b\": container with ID starting with 3f0b72367d407f0525085dcdad4c805480e7236a67e4d757c526ee296e7a803b not found: ID does not exist"
Jul 27 20:32:18.797463 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.797445    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"]
Jul 27 20:32:18.802288 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:18.802268    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-555352-openshift-default-54888b6b6d-wf7rx"]
Jul 27 20:32:19.991799 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:19.991767    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a671ce39-da00-4651-9314-777919d2a7f1" path="/var/lib/kubelet/pods/a671ce39-da00-4651-9314-777919d2a7f1/volumes"
Jul 27 20:32:24.818731 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:24.818695    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"]
Jul 27 20:32:24.819258 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:24.818925    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m" podUID="822479a1-9e4f-4bbf-b712-ee4985917b64" containerName="maas-api" containerID="cri-o://1c5a62c2d0d1bc2a6771f009154730ae33dbc8c4e4152186a8f2eaff2322e638" gracePeriod=30
Jul 27 20:32:25.090913 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.090888    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:32:25.116817 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.116788    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcj2d\" (UniqueName: \"kubernetes.io/projected/822479a1-9e4f-4bbf-b712-ee4985917b64-kube-api-access-mcj2d\") pod \"822479a1-9e4f-4bbf-b712-ee4985917b64\" (UID: \"822479a1-9e4f-4bbf-b712-ee4985917b64\") "
Jul 27 20:32:25.116972 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.116833    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/822479a1-9e4f-4bbf-b712-ee4985917b64-maas-api-tls\") pod \"822479a1-9e4f-4bbf-b712-ee4985917b64\" (UID: \"822479a1-9e4f-4bbf-b712-ee4985917b64\") "
Jul 27 20:32:25.116972 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.116875    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/822479a1-9e4f-4bbf-b712-ee4985917b64-openshift-service-ca\") pod \"822479a1-9e4f-4bbf-b712-ee4985917b64\" (UID: \"822479a1-9e4f-4bbf-b712-ee4985917b64\") "
Jul 27 20:32:25.117337 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.117299    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822479a1-9e4f-4bbf-b712-ee4985917b64-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "822479a1-9e4f-4bbf-b712-ee4985917b64" (UID: "822479a1-9e4f-4bbf-b712-ee4985917b64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:32:25.119208 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.119173    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822479a1-9e4f-4bbf-b712-ee4985917b64-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "822479a1-9e4f-4bbf-b712-ee4985917b64" (UID: "822479a1-9e4f-4bbf-b712-ee4985917b64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:32:25.119333 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.119227    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822479a1-9e4f-4bbf-b712-ee4985917b64-kube-api-access-mcj2d" (OuterVolumeSpecName: "kube-api-access-mcj2d") pod "822479a1-9e4f-4bbf-b712-ee4985917b64" (UID: "822479a1-9e4f-4bbf-b712-ee4985917b64"). InnerVolumeSpecName "kube-api-access-mcj2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:32:25.218162 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.218127    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/822479a1-9e4f-4bbf-b712-ee4985917b64-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:25.218162 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.218156    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcj2d\" (UniqueName: \"kubernetes.io/projected/822479a1-9e4f-4bbf-b712-ee4985917b64-kube-api-access-mcj2d\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:25.218162 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.218166    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/822479a1-9e4f-4bbf-b712-ee4985917b64-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:25.819326 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.819294    2562 generic.go:356] "Generic (PLEG): container finished" podID="822479a1-9e4f-4bbf-b712-ee4985917b64" containerID="1c5a62c2d0d1bc2a6771f009154730ae33dbc8c4e4152186a8f2eaff2322e638" exitCode=0
Jul 27 20:32:25.819766 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.819345    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"
Jul 27 20:32:25.819766 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.819357    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m" event={"ID":"822479a1-9e4f-4bbf-b712-ee4985917b64","Type":"ContainerDied","Data":"1c5a62c2d0d1bc2a6771f009154730ae33dbc8c4e4152186a8f2eaff2322e638"}
Jul 27 20:32:25.819766 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.819405    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m" event={"ID":"822479a1-9e4f-4bbf-b712-ee4985917b64","Type":"ContainerDied","Data":"933fbcaa6d8e40e99730f3665dad54111c0d93d45221be67c88d76155a7a5216"}
Jul 27 20:32:25.819766 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.819430    2562 scope.go:117] "RemoveContainer" containerID="1c5a62c2d0d1bc2a6771f009154730ae33dbc8c4e4152186a8f2eaff2322e638"
Jul 27 20:32:25.829590 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.829573    2562 scope.go:117] "RemoveContainer" containerID="1c5a62c2d0d1bc2a6771f009154730ae33dbc8c4e4152186a8f2eaff2322e638"
Jul 27 20:32:25.829880 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:32:25.829846    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5a62c2d0d1bc2a6771f009154730ae33dbc8c4e4152186a8f2eaff2322e638\": container with ID starting with 1c5a62c2d0d1bc2a6771f009154730ae33dbc8c4e4152186a8f2eaff2322e638 not found: ID does not exist" containerID="1c5a62c2d0d1bc2a6771f009154730ae33dbc8c4e4152186a8f2eaff2322e638"
Jul 27 20:32:25.829953 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.829889    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5a62c2d0d1bc2a6771f009154730ae33dbc8c4e4152186a8f2eaff2322e638"} err="failed to get container status \"1c5a62c2d0d1bc2a6771f009154730ae33dbc8c4e4152186a8f2eaff2322e638\": rpc error: code = NotFound desc = could not find container \"1c5a62c2d0d1bc2a6771f009154730ae33dbc8c4e4152186a8f2eaff2322e638\": container with ID starting with 1c5a62c2d0d1bc2a6771f009154730ae33dbc8c4e4152186a8f2eaff2322e638 not found: ID does not exist"
Jul 27 20:32:25.839996 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.839966    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"]
Jul 27 20:32:25.843800 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.843782    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1517d6-65776bb785-tmq5m"]
Jul 27 20:32:25.992377 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:25.992335    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="822479a1-9e4f-4bbf-b712-ee4985917b64" path="/var/lib/kubelet/pods/822479a1-9e4f-4bbf-b712-ee4985917b64/volumes"
Jul 27 20:32:46.263787 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.263744    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"]
Jul 27 20:32:46.264326 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.264035    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq" podUID="301637fc-59a2-422b-a0d1-95ca7027f450" containerName="maas-api" containerID="cri-o://9e27b47efe29d6b88482a5f9120ffd9094be73f5421c6d62f2a936957110f710" gracePeriod=30
Jul 27 20:32:46.289848 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:32:46.289823    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301637fc_59a2_422b_a0d1_95ca7027f450.slice/crio-9e27b47efe29d6b88482a5f9120ffd9094be73f5421c6d62f2a936957110f710.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 20:32:46.557887 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.557863    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:32:46.636511 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.636465    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/301637fc-59a2-422b-a0d1-95ca7027f450-maas-api-tls\") pod \"301637fc-59a2-422b-a0d1-95ca7027f450\" (UID: \"301637fc-59a2-422b-a0d1-95ca7027f450\") "
Jul 27 20:32:46.636705 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.636542    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb4fh\" (UniqueName: \"kubernetes.io/projected/301637fc-59a2-422b-a0d1-95ca7027f450-kube-api-access-vb4fh\") pod \"301637fc-59a2-422b-a0d1-95ca7027f450\" (UID: \"301637fc-59a2-422b-a0d1-95ca7027f450\") "
Jul 27 20:32:46.636705 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.636566    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/301637fc-59a2-422b-a0d1-95ca7027f450-openshift-service-ca\") pod \"301637fc-59a2-422b-a0d1-95ca7027f450\" (UID: \"301637fc-59a2-422b-a0d1-95ca7027f450\") "
Jul 27 20:32:46.636978 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.636955    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/301637fc-59a2-422b-a0d1-95ca7027f450-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "301637fc-59a2-422b-a0d1-95ca7027f450" (UID: "301637fc-59a2-422b-a0d1-95ca7027f450"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:32:46.638674 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.638649    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/301637fc-59a2-422b-a0d1-95ca7027f450-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "301637fc-59a2-422b-a0d1-95ca7027f450" (UID: "301637fc-59a2-422b-a0d1-95ca7027f450"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:32:46.638674 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.638669    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301637fc-59a2-422b-a0d1-95ca7027f450-kube-api-access-vb4fh" (OuterVolumeSpecName: "kube-api-access-vb4fh") pod "301637fc-59a2-422b-a0d1-95ca7027f450" (UID: "301637fc-59a2-422b-a0d1-95ca7027f450"). InnerVolumeSpecName "kube-api-access-vb4fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:32:46.738378 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.738344    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/301637fc-59a2-422b-a0d1-95ca7027f450-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:46.738378 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.738371    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb4fh\" (UniqueName: \"kubernetes.io/projected/301637fc-59a2-422b-a0d1-95ca7027f450-kube-api-access-vb4fh\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:46.738378 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.738381    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/301637fc-59a2-422b-a0d1-95ca7027f450-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:32:46.936582 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.936550    2562 generic.go:356] "Generic (PLEG): container finished" podID="301637fc-59a2-422b-a0d1-95ca7027f450" containerID="9e27b47efe29d6b88482a5f9120ffd9094be73f5421c6d62f2a936957110f710" exitCode=0
Jul 27 20:32:46.936791 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.936610    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"
Jul 27 20:32:46.936791 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.936610    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq" event={"ID":"301637fc-59a2-422b-a0d1-95ca7027f450","Type":"ContainerDied","Data":"9e27b47efe29d6b88482a5f9120ffd9094be73f5421c6d62f2a936957110f710"}
Jul 27 20:32:46.936791 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.936676    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq" event={"ID":"301637fc-59a2-422b-a0d1-95ca7027f450","Type":"ContainerDied","Data":"ea1866ef24c33faf6bf7ce478d5c08a321a3e9afa5325217243bcae39774bcd8"}
Jul 27 20:32:46.936791 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.936707    2562 scope.go:117] "RemoveContainer" containerID="9e27b47efe29d6b88482a5f9120ffd9094be73f5421c6d62f2a936957110f710"
Jul 27 20:32:46.946917 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.946894    2562 scope.go:117] "RemoveContainer" containerID="9e27b47efe29d6b88482a5f9120ffd9094be73f5421c6d62f2a936957110f710"
Jul 27 20:32:46.947210 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:32:46.947186    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e27b47efe29d6b88482a5f9120ffd9094be73f5421c6d62f2a936957110f710\": container with ID starting with 9e27b47efe29d6b88482a5f9120ffd9094be73f5421c6d62f2a936957110f710 not found: ID does not exist" containerID="9e27b47efe29d6b88482a5f9120ffd9094be73f5421c6d62f2a936957110f710"
Jul 27 20:32:46.947267 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.947225    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e27b47efe29d6b88482a5f9120ffd9094be73f5421c6d62f2a936957110f710"} err="failed to get container status \"9e27b47efe29d6b88482a5f9120ffd9094be73f5421c6d62f2a936957110f710\": rpc error: code = NotFound desc = could not find container \"9e27b47efe29d6b88482a5f9120ffd9094be73f5421c6d62f2a936957110f710\": container with ID starting with 9e27b47efe29d6b88482a5f9120ffd9094be73f5421c6d62f2a936957110f710 not found: ID does not exist"
Jul 27 20:32:46.959067 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.959041    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"]
Jul 27 20:32:46.962406 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:46.962379    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c3a53e-696ccf899b-tfzpq"]
Jul 27 20:32:47.997103 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:47.997064    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="301637fc-59a2-422b-a0d1-95ca7027f450" path="/var/lib/kubelet/pods/301637fc-59a2-422b-a0d1-95ca7027f450/volumes"
Jul 27 20:32:57.464579 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:57.464528    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"]
Jul 27 20:32:57.465108 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:32:57.464806    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6" podUID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerName="istio-proxy" containerID="cri-o://b47f1097c562a899be22823e2b637d8078e550f7dd5c05ddd31bf58084684a3a" gracePeriod=30
Jul 27 20:33:02.718297 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.718269    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:33:02.803775 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.803677    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-token\") pod \"bf8d180a-511d-4e57-af8a-40be2ff7c277\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") "
Jul 27 20:33:02.803775 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.803719    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-data\") pod \"bf8d180a-511d-4e57-af8a-40be2ff7c277\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") "
Jul 27 20:33:02.804118 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.803851    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-credential-socket\") pod \"bf8d180a-511d-4e57-af8a-40be2ff7c277\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") "
Jul 27 20:33:02.804118 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.803921    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-podinfo\") pod \"bf8d180a-511d-4e57-af8a-40be2ff7c277\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") "
Jul 27 20:33:02.804118 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.803950    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-envoy\") pod \"bf8d180a-511d-4e57-af8a-40be2ff7c277\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") "
Jul 27 20:33:02.804118 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.803987    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-workload-socket\") pod \"bf8d180a-511d-4e57-af8a-40be2ff7c277\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") "
Jul 27 20:33:02.804118 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.804033    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhr4t\" (UniqueName: \"kubernetes.io/projected/bf8d180a-511d-4e57-af8a-40be2ff7c277-kube-api-access-dhr4t\") pod \"bf8d180a-511d-4e57-af8a-40be2ff7c277\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") "
Jul 27 20:33:02.804118 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.804092    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8d180a-511d-4e57-af8a-40be2ff7c277-istiod-ca-cert\") pod \"bf8d180a-511d-4e57-af8a-40be2ff7c277\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") "
Jul 27 20:33:02.804118 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.804119    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-workload-certs\") pod \"bf8d180a-511d-4e57-af8a-40be2ff7c277\" (UID: \"bf8d180a-511d-4e57-af8a-40be2ff7c277\") "
Jul 27 20:33:02.804511 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.804209    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bf8d180a-511d-4e57-af8a-40be2ff7c277" (UID: "bf8d180a-511d-4e57-af8a-40be2ff7c277"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:33:02.804610 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.804588    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-credential-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:02.804743 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.804608    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-data" (OuterVolumeSpecName: "istio-data") pod "bf8d180a-511d-4e57-af8a-40be2ff7c277" (UID: "bf8d180a-511d-4e57-af8a-40be2ff7c277"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:33:02.804743 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.804691    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bf8d180a-511d-4e57-af8a-40be2ff7c277" (UID: "bf8d180a-511d-4e57-af8a-40be2ff7c277"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:33:02.804858 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.804838    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bf8d180a-511d-4e57-af8a-40be2ff7c277" (UID: "bf8d180a-511d-4e57-af8a-40be2ff7c277"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:33:02.804974 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.804950    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8d180a-511d-4e57-af8a-40be2ff7c277-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bf8d180a-511d-4e57-af8a-40be2ff7c277" (UID: "bf8d180a-511d-4e57-af8a-40be2ff7c277"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:33:02.806303 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.806272    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bf8d180a-511d-4e57-af8a-40be2ff7c277" (UID: "bf8d180a-511d-4e57-af8a-40be2ff7c277"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:33:02.806761 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.806740    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf8d180a-511d-4e57-af8a-40be2ff7c277" (UID: "bf8d180a-511d-4e57-af8a-40be2ff7c277"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:33:02.806839 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.806764    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bf8d180a-511d-4e57-af8a-40be2ff7c277" (UID: "bf8d180a-511d-4e57-af8a-40be2ff7c277"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:33:02.806934 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.806904    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8d180a-511d-4e57-af8a-40be2ff7c277-kube-api-access-dhr4t" (OuterVolumeSpecName: "kube-api-access-dhr4t") pod "bf8d180a-511d-4e57-af8a-40be2ff7c277" (UID: "bf8d180a-511d-4e57-af8a-40be2ff7c277"). InnerVolumeSpecName "kube-api-access-dhr4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:33:02.906059 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.906023    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-podinfo\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:02.906059 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.906050    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-envoy\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:02.906059 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.906060    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-workload-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:02.906059 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.906069    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhr4t\" (UniqueName: \"kubernetes.io/projected/bf8d180a-511d-4e57-af8a-40be2ff7c277-kube-api-access-dhr4t\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:02.906328 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.906077    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8d180a-511d-4e57-af8a-40be2ff7c277-istiod-ca-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:02.906328 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.906087    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-workload-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:02.906328 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.906095    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:02.906328 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:02.906102    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8d180a-511d-4e57-af8a-40be2ff7c277-istio-data\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:03.030776 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:03.030742    2562 generic.go:356] "Generic (PLEG): container finished" podID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerID="b47f1097c562a899be22823e2b637d8078e550f7dd5c05ddd31bf58084684a3a" exitCode=0
Jul 27 20:33:03.030937 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:03.030791    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"
Jul 27 20:33:03.030937 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:03.030824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6" event={"ID":"bf8d180a-511d-4e57-af8a-40be2ff7c277","Type":"ContainerDied","Data":"b47f1097c562a899be22823e2b637d8078e550f7dd5c05ddd31bf58084684a3a"}
Jul 27 20:33:03.030937 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:03.030872    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6" event={"ID":"bf8d180a-511d-4e57-af8a-40be2ff7c277","Type":"ContainerDied","Data":"a842ef623077658240c3de78c4d21c1f68bd4b134639bd4b86d8f1efead0b241"}
Jul 27 20:33:03.030937 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:03.030892    2562 scope.go:117] "RemoveContainer" containerID="b47f1097c562a899be22823e2b637d8078e550f7dd5c05ddd31bf58084684a3a"
Jul 27 20:33:03.040793 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:03.040776    2562 scope.go:117] "RemoveContainer" containerID="bc25defe80a660f8ad34cf1e0ff1abab5e06213d2610ac6aae85c4ba5741ede2"
Jul 27 20:33:03.049705 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:03.049686    2562 scope.go:117] "RemoveContainer" containerID="b47f1097c562a899be22823e2b637d8078e550f7dd5c05ddd31bf58084684a3a"
Jul 27 20:33:03.050042 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:33:03.050017    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47f1097c562a899be22823e2b637d8078e550f7dd5c05ddd31bf58084684a3a\": container with ID starting with b47f1097c562a899be22823e2b637d8078e550f7dd5c05ddd31bf58084684a3a not found: ID does not exist" containerID="b47f1097c562a899be22823e2b637d8078e550f7dd5c05ddd31bf58084684a3a"
Jul 27 20:33:03.050116 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:03.050055    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47f1097c562a899be22823e2b637d8078e550f7dd5c05ddd31bf58084684a3a"} err="failed to get container status \"b47f1097c562a899be22823e2b637d8078e550f7dd5c05ddd31bf58084684a3a\": rpc error: code = NotFound desc = could not find container \"b47f1097c562a899be22823e2b637d8078e550f7dd5c05ddd31bf58084684a3a\": container with ID starting with b47f1097c562a899be22823e2b637d8078e550f7dd5c05ddd31bf58084684a3a not found: ID does not exist"
Jul 27 20:33:03.050116 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:03.050081    2562 scope.go:117] "RemoveContainer" containerID="bc25defe80a660f8ad34cf1e0ff1abab5e06213d2610ac6aae85c4ba5741ede2"
Jul 27 20:33:03.050378 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:33:03.050359    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc25defe80a660f8ad34cf1e0ff1abab5e06213d2610ac6aae85c4ba5741ede2\": container with ID starting with bc25defe80a660f8ad34cf1e0ff1abab5e06213d2610ac6aae85c4ba5741ede2 not found: ID does not exist" containerID="bc25defe80a660f8ad34cf1e0ff1abab5e06213d2610ac6aae85c4ba5741ede2"
Jul 27 20:33:03.050440 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:03.050387    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc25defe80a660f8ad34cf1e0ff1abab5e06213d2610ac6aae85c4ba5741ede2"} err="failed to get container status \"bc25defe80a660f8ad34cf1e0ff1abab5e06213d2610ac6aae85c4ba5741ede2\": rpc error: code = NotFound desc = could not find container \"bc25defe80a660f8ad34cf1e0ff1abab5e06213d2610ac6aae85c4ba5741ede2\": container with ID starting with bc25defe80a660f8ad34cf1e0ff1abab5e06213d2610ac6aae85c4ba5741ede2 not found: ID does not exist"
Jul 27 20:33:03.052087 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:03.052068    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"]
Jul 27 20:33:03.054967 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:03.054919    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-c3a53e-openshift-default-59b5844548-l7lt6"]
Jul 27 20:33:03.992436 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:03.992403    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf8d180a-511d-4e57-af8a-40be2ff7c277" path="/var/lib/kubelet/pods/bf8d180a-511d-4e57-af8a-40be2ff7c277/volumes"
Jul 27 20:33:09.265862 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:09.264884    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"]
Jul 27 20:33:09.265862 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:09.265143    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk" podUID="0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc" containerName="maas-api" containerID="cri-o://ad81419c38f8d58fb62f3d44c95eb4ecbe1cc71e95b1f93a8b621b62ad34eece" gracePeriod=30
Jul 27 20:33:09.543859 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:09.543834    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:33:09.675435 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:09.675400    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hf9rg\" (UniqueName: \"kubernetes.io/projected/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-kube-api-access-hf9rg\") pod \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\" (UID: \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\") "
Jul 27 20:33:09.675818 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:09.675793    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-openshift-service-ca\") pod \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\" (UID: \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\") "
Jul 27 20:33:09.675944 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:09.675846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-maas-api-tls\") pod \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\" (UID: \"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc\") "
Jul 27 20:33:09.676171 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:09.676144    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc" (UID: "0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:33:09.677911 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:09.677882    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-kube-api-access-hf9rg" (OuterVolumeSpecName: "kube-api-access-hf9rg") pod "0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc" (UID: "0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc"). InnerVolumeSpecName "kube-api-access-hf9rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:33:09.678328 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:09.678305    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc" (UID: "0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:33:09.777176 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:09.777082    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-openshift-service-ca\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:09.777176 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:09.777125    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-maas-api-tls\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:09.777176 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:09.777142    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf9rg\" (UniqueName: \"kubernetes.io/projected/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc-kube-api-access-hf9rg\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:10.069890 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:10.069801    2562 generic.go:356] "Generic (PLEG): container finished" podID="0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc" containerID="ad81419c38f8d58fb62f3d44c95eb4ecbe1cc71e95b1f93a8b621b62ad34eece" exitCode=0
Jul 27 20:33:10.069890 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:10.069846    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"
Jul 27 20:33:10.070068 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:10.069890    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk" event={"ID":"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc","Type":"ContainerDied","Data":"ad81419c38f8d58fb62f3d44c95eb4ecbe1cc71e95b1f93a8b621b62ad34eece"}
Jul 27 20:33:10.070068 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:10.069949    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk" event={"ID":"0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc","Type":"ContainerDied","Data":"342b1e00b0c2bbba3a8e7b7dc406bb26d4a64af755f3511a7cfa6d01141ad970"}
Jul 27 20:33:10.070068 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:10.069973    2562 scope.go:117] "RemoveContainer" containerID="ad81419c38f8d58fb62f3d44c95eb4ecbe1cc71e95b1f93a8b621b62ad34eece"
Jul 27 20:33:10.080406 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:10.080371    2562 scope.go:117] "RemoveContainer" containerID="ad81419c38f8d58fb62f3d44c95eb4ecbe1cc71e95b1f93a8b621b62ad34eece"
Jul 27 20:33:10.080710 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:33:10.080689    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad81419c38f8d58fb62f3d44c95eb4ecbe1cc71e95b1f93a8b621b62ad34eece\": container with ID starting with ad81419c38f8d58fb62f3d44c95eb4ecbe1cc71e95b1f93a8b621b62ad34eece not found: ID does not exist" containerID="ad81419c38f8d58fb62f3d44c95eb4ecbe1cc71e95b1f93a8b621b62ad34eece"
Jul 27 20:33:10.080810 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:10.080718    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad81419c38f8d58fb62f3d44c95eb4ecbe1cc71e95b1f93a8b621b62ad34eece"} err="failed to get container status \"ad81419c38f8d58fb62f3d44c95eb4ecbe1cc71e95b1f93a8b621b62ad34eece\": rpc error: code = NotFound desc = could not find container \"ad81419c38f8d58fb62f3d44c95eb4ecbe1cc71e95b1f93a8b621b62ad34eece\": container with ID starting with ad81419c38f8d58fb62f3d44c95eb4ecbe1cc71e95b1f93a8b621b62ad34eece not found: ID does not exist"
Jul 27 20:33:10.085059 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:10.085039    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"]
Jul 27 20:33:10.086989 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:10.086969    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f44149-6998fdcf45-cj6hk"]
Jul 27 20:33:11.991934 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:11.991899    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc" path="/var/lib/kubelet/pods/0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc/volumes"
Jul 27 20:33:19.472096 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:19.472064    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"]
Jul 27 20:33:19.472596 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:19.472271    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw" podUID="2eb244db-e1bc-4802-9e2a-69a5de755d51" containerName="istio-proxy" containerID="cri-o://c45d6fba40a2f9f4efcae25719f559d82fbbdf06fea4351f5f167fd08908ff6a" gracePeriod=30
Jul 27 20:33:20.333378 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:20.333340    2562 patch_prober.go:28] interesting pod/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 20:33:20.333688 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:20.333403    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw" podUID="2eb244db-e1bc-4802-9e2a-69a5de755d51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 20:33:24.730374 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.730344    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:33:24.822623 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.822531    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-token\") pod \"2eb244db-e1bc-4802-9e2a-69a5de755d51\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") "
Jul 27 20:33:24.822623 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.822585    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pngn\" (UniqueName: \"kubernetes.io/projected/2eb244db-e1bc-4802-9e2a-69a5de755d51-kube-api-access-7pngn\") pod \"2eb244db-e1bc-4802-9e2a-69a5de755d51\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") "
Jul 27 20:33:24.822623 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.822617    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-data\") pod \"2eb244db-e1bc-4802-9e2a-69a5de755d51\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") "
Jul 27 20:33:24.822908 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.822725    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-podinfo\") pod \"2eb244db-e1bc-4802-9e2a-69a5de755d51\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") "
Jul 27 20:33:24.822908 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.822791    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eb244db-e1bc-4802-9e2a-69a5de755d51-istiod-ca-cert\") pod \"2eb244db-e1bc-4802-9e2a-69a5de755d51\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") "
Jul 27 20:33:24.822908 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.822830    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-envoy\") pod \"2eb244db-e1bc-4802-9e2a-69a5de755d51\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") "
Jul 27 20:33:24.822908 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.822852    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-workload-socket\") pod \"2eb244db-e1bc-4802-9e2a-69a5de755d51\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") "
Jul 27 20:33:24.822908 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.822899    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-workload-certs\") pod \"2eb244db-e1bc-4802-9e2a-69a5de755d51\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") "
Jul 27 20:33:24.823177 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.822933    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-credential-socket\") pod \"2eb244db-e1bc-4802-9e2a-69a5de755d51\" (UID: \"2eb244db-e1bc-4802-9e2a-69a5de755d51\") "
Jul 27 20:33:24.823177 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.823152    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2eb244db-e1bc-4802-9e2a-69a5de755d51" (UID: "2eb244db-e1bc-4802-9e2a-69a5de755d51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:33:24.823291 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.823169    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2eb244db-e1bc-4802-9e2a-69a5de755d51" (UID: "2eb244db-e1bc-4802-9e2a-69a5de755d51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:33:24.823542 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.823355    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2eb244db-e1bc-4802-9e2a-69a5de755d51" (UID: "2eb244db-e1bc-4802-9e2a-69a5de755d51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:33:24.823542 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.823533    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-workload-certs\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:24.823665 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.823538    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-data" (OuterVolumeSpecName: "istio-data") pod "2eb244db-e1bc-4802-9e2a-69a5de755d51" (UID: "2eb244db-e1bc-4802-9e2a-69a5de755d51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:33:24.823665 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.823554    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-workload-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:24.823665 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.823565    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb244db-e1bc-4802-9e2a-69a5de755d51-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2eb244db-e1bc-4802-9e2a-69a5de755d51" (UID: "2eb244db-e1bc-4802-9e2a-69a5de755d51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:33:24.825065 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.825040    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-token" (OuterVolumeSpecName: "istio-token") pod "2eb244db-e1bc-4802-9e2a-69a5de755d51" (UID: "2eb244db-e1bc-4802-9e2a-69a5de755d51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:33:24.825065 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.825054    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb244db-e1bc-4802-9e2a-69a5de755d51-kube-api-access-7pngn" (OuterVolumeSpecName: "kube-api-access-7pngn") pod "2eb244db-e1bc-4802-9e2a-69a5de755d51" (UID: "2eb244db-e1bc-4802-9e2a-69a5de755d51"). InnerVolumeSpecName "kube-api-access-7pngn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:33:24.825229 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.825061    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2eb244db-e1bc-4802-9e2a-69a5de755d51" (UID: "2eb244db-e1bc-4802-9e2a-69a5de755d51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:33:24.825398 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.825377    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2eb244db-e1bc-4802-9e2a-69a5de755d51" (UID: "2eb244db-e1bc-4802-9e2a-69a5de755d51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:33:24.924716 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.924676    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-envoy\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:24.924716 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.924714    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-credential-socket\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:24.924927 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.924730    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-token\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:24.924927 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.924745    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pngn\" (UniqueName: \"kubernetes.io/projected/2eb244db-e1bc-4802-9e2a-69a5de755d51-kube-api-access-7pngn\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:24.924927 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.924760    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-data\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:24.924927 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.924773    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eb244db-e1bc-4802-9e2a-69a5de755d51-istio-podinfo\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:24.924927 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:24.924786    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eb244db-e1bc-4802-9e2a-69a5de755d51-istiod-ca-cert\") on node \"ip-10-0-142-140.ec2.internal\" DevicePath \"\""
Jul 27 20:33:25.150937 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.150902    2562 generic.go:356] "Generic (PLEG): container finished" podID="2eb244db-e1bc-4802-9e2a-69a5de755d51" containerID="c45d6fba40a2f9f4efcae25719f559d82fbbdf06fea4351f5f167fd08908ff6a" exitCode=0
Jul 27 20:33:25.151134 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.150954    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"
Jul 27 20:33:25.151134 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.150989    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw" event={"ID":"2eb244db-e1bc-4802-9e2a-69a5de755d51","Type":"ContainerDied","Data":"c45d6fba40a2f9f4efcae25719f559d82fbbdf06fea4351f5f167fd08908ff6a"}
Jul 27 20:33:25.151134 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.151056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw" event={"ID":"2eb244db-e1bc-4802-9e2a-69a5de755d51","Type":"ContainerDied","Data":"d602461236875b14c8773c9fd0c31f025759c24c9142892ec189b83a93ed10f0"}
Jul 27 20:33:25.151134 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.151091    2562 scope.go:117] "RemoveContainer" containerID="c45d6fba40a2f9f4efcae25719f559d82fbbdf06fea4351f5f167fd08908ff6a"
Jul 27 20:33:25.160553 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.160537    2562 scope.go:117] "RemoveContainer" containerID="062e69e32e0fee38acdae45c2f9f9024da3a632e3f70eb6150384992c85776bb"
Jul 27 20:33:25.168531 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.168513    2562 scope.go:117] "RemoveContainer" containerID="c45d6fba40a2f9f4efcae25719f559d82fbbdf06fea4351f5f167fd08908ff6a"
Jul 27 20:33:25.168768 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:33:25.168749    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c45d6fba40a2f9f4efcae25719f559d82fbbdf06fea4351f5f167fd08908ff6a\": container with ID starting with c45d6fba40a2f9f4efcae25719f559d82fbbdf06fea4351f5f167fd08908ff6a not found: ID does not exist" containerID="c45d6fba40a2f9f4efcae25719f559d82fbbdf06fea4351f5f167fd08908ff6a"
Jul 27 20:33:25.168829 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.168777    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45d6fba40a2f9f4efcae25719f559d82fbbdf06fea4351f5f167fd08908ff6a"} err="failed to get container status \"c45d6fba40a2f9f4efcae25719f559d82fbbdf06fea4351f5f167fd08908ff6a\": rpc error: code = NotFound desc = could not find container \"c45d6fba40a2f9f4efcae25719f559d82fbbdf06fea4351f5f167fd08908ff6a\": container with ID starting with c45d6fba40a2f9f4efcae25719f559d82fbbdf06fea4351f5f167fd08908ff6a not found: ID does not exist"
Jul 27 20:33:25.168829 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.168795    2562 scope.go:117] "RemoveContainer" containerID="062e69e32e0fee38acdae45c2f9f9024da3a632e3f70eb6150384992c85776bb"
Jul 27 20:33:25.169040 ip-10-0-142-140 kubenswrapper[2562]: E0727 20:33:25.169019    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"062e69e32e0fee38acdae45c2f9f9024da3a632e3f70eb6150384992c85776bb\": container with ID starting with 062e69e32e0fee38acdae45c2f9f9024da3a632e3f70eb6150384992c85776bb not found: ID does not exist" containerID="062e69e32e0fee38acdae45c2f9f9024da3a632e3f70eb6150384992c85776bb"
Jul 27 20:33:25.169097 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.169055    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"062e69e32e0fee38acdae45c2f9f9024da3a632e3f70eb6150384992c85776bb"} err="failed to get container status \"062e69e32e0fee38acdae45c2f9f9024da3a632e3f70eb6150384992c85776bb\": rpc error: code = NotFound desc = could not find container \"062e69e32e0fee38acdae45c2f9f9024da3a632e3f70eb6150384992c85776bb\": container with ID starting with 062e69e32e0fee38acdae45c2f9f9024da3a632e3f70eb6150384992c85776bb not found: ID does not exist"
Jul 27 20:33:25.172351 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.172326    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"]
Jul 27 20:33:25.175338 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.175318    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-f44149-openshift-default-5585689fb4-gxcbw"]
Jul 27 20:33:25.422516 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.422421    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pxkfm_b4cd6b2f-c1db-4267-a123-ba564753c91d/manager/0.log"
Jul 27 20:33:25.528430 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.528400    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bwq5w_21a0d151-3240-46fe-8c19-e64e2d60e684/manager/0.log"
Jul 27 20:33:25.642568 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.642533    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8b889c85-k7rvz_ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5/manager/0.log"
Jul 27 20:33:25.761056 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.760976    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-shdxq_a7ed7694-6b8f-4519-bc40-9ac72f89945a/server/0.log"
Jul 27 20:33:25.875739 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.875714    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-p79zv_a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa/manager/0.log"
Jul 27 20:33:25.982085 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.982055    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df5588f67-626t9_56f55834-fe4e-403f-a5ad-1d21555238a9/copy-manifests/0.log"
Jul 27 20:33:25.990148 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.990124    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df5588f67-626t9_56f55834-fe4e-403f-a5ad-1d21555238a9/manager/0.log"
Jul 27 20:33:25.991657 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:25.991635    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eb244db-e1bc-4802-9e2a-69a5de755d51" path="/var/lib/kubelet/pods/2eb244db-e1bc-4802-9e2a-69a5de755d51/volumes"
Jul 27 20:33:26.597456 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:26.597425    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-77b4b4d988-h85l4_60115a3c-980a-4762-a08d-40532cc0c63c/maas-api/0.log"
Jul 27 20:33:26.709679 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:26.709655    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753040-p8qqd_6604a889-4fb4-4402-9b7e-3452eb63b2a8/cleanup/0.log"
Jul 27 20:33:27.030362 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:27.030299    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-w6fbk_72b8cb4d-eae2-4786-853a-de7f919e38d7/postgres/0.log"
Jul 27 20:33:28.379208 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:28.379165    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-8hvxh_6c006c97-5e13-4121-9ae8-433fdbae3781/manager/0.log"
Jul 27 20:33:28.488641 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:28.488614    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-dgvzl_f9938b9c-d76a-4c14-b8e1-95dad0295fea/manager/0.log"
Jul 27 20:33:28.593906 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:28.593871    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-f6kfh_12c1ad92-cf64-47f8-aff2-3f1a9aebab7d/kuadrant-console-plugin/0.log"
Jul 27 20:33:28.706131 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:28.706057    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-pvj4x_8abd6d76-25d6-4667-a7d2-051a753e8972/registry-server/0.log"
Jul 27 20:33:28.820009 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:28.819977    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dnrwh_2d7fce82-17ec-4d6c-ae75-c76cdafcd56b/manager/0.log"
Jul 27 20:33:29.050190 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:29.050117    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hx725_6fa4c9e5-3210-49bb-a40a-f14b21d5af13/manager/0.log"
Jul 27 20:33:29.383204 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:29.383178    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw_64c987d6-44b0-4d39-8bb4-a4e5cc37e157/istio-proxy/0.log"
Jul 27 20:33:29.722434 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:29.722361    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fd5d97dbc-fllhj_8799eeb6-ba19-46de-85e6-007eb62dcf70/kube-auth-proxy/0.log"
Jul 27 20:33:29.842742 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:29.842711    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vbkhz_861d7f38-16d8-41ed-bf20-d15a7bcc6cbf/istio-proxy/1.log"
Jul 27 20:33:29.958216 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:29.958187    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-952m8_6da65f36-1bc2-44a6-b6f7-52cf19b524d6/payload-pre-processing/0.log"
Jul 27 20:33:30.071618 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:30.071513    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wrscr_65bda14a-6ab1-406e-8754-305218a633ab/payload-processing/0.log"
Jul 27 20:33:30.507451 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:30.507416    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-tmdvz_b9e9e33b-ea90-4d81-8285-20dc7e7bea67/main/0.log"
Jul 27 20:33:36.334566 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.334534    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sgjdf/must-gather-jxxzb"]
Jul 27 20:33:36.334955 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.334942    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0aab554-59e6-4908-8d29-18ba81008ebc" containerName="maas-api"
Jul 27 20:33:36.334955 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.334954    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0aab554-59e6-4908-8d29-18ba81008ebc" containerName="maas-api"
Jul 27 20:33:36.335031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.334965    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb244db-e1bc-4802-9e2a-69a5de755d51" containerName="istio-proxy"
Jul 27 20:33:36.335031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.334971    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb244db-e1bc-4802-9e2a-69a5de755d51" containerName="istio-proxy"
Jul 27 20:33:36.335031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.334979    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerName="istio-proxy"
Jul 27 20:33:36.335031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.334985    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerName="istio-proxy"
Jul 27 20:33:36.335031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.334996    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb244db-e1bc-4802-9e2a-69a5de755d51" containerName="istio-proxy"
Jul 27 20:33:36.335031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335002    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb244db-e1bc-4802-9e2a-69a5de755d51" containerName="istio-proxy"
Jul 27 20:33:36.335031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335008    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3bdbbac-3b26-4549-a4b5-82a997f3f579" containerName="maas-api"
Jul 27 20:33:36.335031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335013    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3bdbbac-3b26-4549-a4b5-82a997f3f579" containerName="maas-api"
Jul 27 20:33:36.335031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335021    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc" containerName="maas-api"
Jul 27 20:33:36.335031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335025    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc" containerName="maas-api"
Jul 27 20:33:36.335031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335035    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822479a1-9e4f-4bbf-b712-ee4985917b64" containerName="maas-api"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335040    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="822479a1-9e4f-4bbf-b712-ee4985917b64" containerName="maas-api"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335048    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a671ce39-da00-4651-9314-777919d2a7f1" containerName="istio-proxy"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335053    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a671ce39-da00-4651-9314-777919d2a7f1" containerName="istio-proxy"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335064    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="301637fc-59a2-422b-a0d1-95ca7027f450" containerName="maas-api"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335075    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="301637fc-59a2-422b-a0d1-95ca7027f450" containerName="maas-api"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335092    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a671ce39-da00-4651-9314-777919d2a7f1" containerName="istio-proxy"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335100    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a671ce39-da00-4651-9314-777919d2a7f1" containerName="istio-proxy"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335179    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerName="istio-proxy"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335193    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3bdbbac-3b26-4549-a4b5-82a997f3f579" containerName="maas-api"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335204    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0aab554-59e6-4908-8d29-18ba81008ebc" containerName="maas-api"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335212    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerName="istio-proxy"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335221    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="301637fc-59a2-422b-a0d1-95ca7027f450" containerName="maas-api"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335231    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb244db-e1bc-4802-9e2a-69a5de755d51" containerName="istio-proxy"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335240    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a671ce39-da00-4651-9314-777919d2a7f1" containerName="istio-proxy"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335248    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb244db-e1bc-4802-9e2a-69a5de755d51" containerName="istio-proxy"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335256    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a671ce39-da00-4651-9314-777919d2a7f1" containerName="istio-proxy"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335263    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="822479a1-9e4f-4bbf-b712-ee4985917b64" containerName="maas-api"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335272    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da2b52f-fc77-4e9f-8be3-3d45bfeaedcc" containerName="maas-api"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335390    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerName="istio-proxy"
Jul 27 20:33:36.335589 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.335404    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf8d180a-511d-4e57-af8a-40be2ff7c277" containerName="istio-proxy"
Jul 27 20:33:36.338792 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.338773    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sgjdf/must-gather-jxxzb"
Jul 27 20:33:36.340825 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.340802    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sgjdf\"/\"kube-root-ca.crt\""
Jul 27 20:33:36.340934 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.340843    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sgjdf\"/\"openshift-service-ca.crt\""
Jul 27 20:33:36.346546 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.346525    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sgjdf/must-gather-jxxzb"]
Jul 27 20:33:36.435680 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.435645    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/80c6ec07-dd99-46da-a370-0ce3d637262f-must-gather-output\") pod \"must-gather-jxxzb\" (UID: \"80c6ec07-dd99-46da-a370-0ce3d637262f\") " pod="openshift-must-gather-sgjdf/must-gather-jxxzb"
Jul 27 20:33:36.435864 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.435711    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrrs2\" (UniqueName: \"kubernetes.io/projected/80c6ec07-dd99-46da-a370-0ce3d637262f-kube-api-access-jrrs2\") pod \"must-gather-jxxzb\" (UID: \"80c6ec07-dd99-46da-a370-0ce3d637262f\") " pod="openshift-must-gather-sgjdf/must-gather-jxxzb"
Jul 27 20:33:36.536878 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.536842    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/80c6ec07-dd99-46da-a370-0ce3d637262f-must-gather-output\") pod \"must-gather-jxxzb\" (UID: \"80c6ec07-dd99-46da-a370-0ce3d637262f\") " pod="openshift-must-gather-sgjdf/must-gather-jxxzb"
Jul 27 20:33:36.537044 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.536913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrrs2\" (UniqueName: \"kubernetes.io/projected/80c6ec07-dd99-46da-a370-0ce3d637262f-kube-api-access-jrrs2\") pod \"must-gather-jxxzb\" (UID: \"80c6ec07-dd99-46da-a370-0ce3d637262f\") " pod="openshift-must-gather-sgjdf/must-gather-jxxzb"
Jul 27 20:33:36.537157 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.537140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/80c6ec07-dd99-46da-a370-0ce3d637262f-must-gather-output\") pod \"must-gather-jxxzb\" (UID: \"80c6ec07-dd99-46da-a370-0ce3d637262f\") " pod="openshift-must-gather-sgjdf/must-gather-jxxzb"
Jul 27 20:33:36.544800 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.544779    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrrs2\" (UniqueName: \"kubernetes.io/projected/80c6ec07-dd99-46da-a370-0ce3d637262f-kube-api-access-jrrs2\") pod \"must-gather-jxxzb\" (UID: \"80c6ec07-dd99-46da-a370-0ce3d637262f\") " pod="openshift-must-gather-sgjdf/must-gather-jxxzb"
Jul 27 20:33:36.648768 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.648739    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sgjdf/must-gather-jxxzb"
Jul 27 20:33:36.809612 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.809585    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sgjdf/must-gather-jxxzb"]
Jul 27 20:33:36.811935 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:33:36.811909    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80c6ec07_dd99_46da_a370_0ce3d637262f.slice/crio-f134106ac5b9653181b322a351ed851351129e5b643163382b4632cba830187b WatchSource:0}: Error finding container f134106ac5b9653181b322a351ed851351129e5b643163382b4632cba830187b: Status 404 returned error can't find the container with id f134106ac5b9653181b322a351ed851351129e5b643163382b4632cba830187b
Jul 27 20:33:36.813746 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:36.813730    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:33:37.218644 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:37.218597    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sgjdf/must-gather-jxxzb" event={"ID":"80c6ec07-dd99-46da-a370-0ce3d637262f","Type":"ContainerStarted","Data":"f134106ac5b9653181b322a351ed851351129e5b643163382b4632cba830187b"}
Jul 27 20:33:38.228979 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:38.228938    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sgjdf/must-gather-jxxzb" event={"ID":"80c6ec07-dd99-46da-a370-0ce3d637262f","Type":"ContainerStarted","Data":"5877c35b22ed05a655649c41970ca2e4f444a33a2bbae8a6482d8c398616b03e"}
Jul 27 20:33:38.229452 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:38.228993    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sgjdf/must-gather-jxxzb" event={"ID":"80c6ec07-dd99-46da-a370-0ce3d637262f","Type":"ContainerStarted","Data":"7f77f29289aa2fd19451925d413bdc5714982f018865932af8dd123d45d906d0"}
Jul 27 20:33:38.245525 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:38.245452    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sgjdf/must-gather-jxxzb" podStartSLOduration=1.399132878 podStartE2EDuration="2.245436855s" totalImagesPullingTime="846.303977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:33:38.244095278 +0000 UTC m=+3232.796011696" watchObservedRunningTime="2026-07-27 20:33:38.245436855 +0000 UTC m=+3232.797353273"
Jul 27 20:33:39.293117 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:39.293080    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jbl6q_003428df-67c2-4b3c-906a-260c8728b23a/global-pull-secret-syncer/0.log"
Jul 27 20:33:39.442910 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:39.442880    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x5d7t_af41025b-248c-4634-8c46-68d227fa3478/konnectivity-agent/0.log"
Jul 27 20:33:39.513387 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:39.513351    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-140.ec2.internal_028ba8eadc770529daa4c0f4fe338d90/haproxy/0.log"
Jul 27 20:33:44.574360 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:44.574328    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-8hvxh_6c006c97-5e13-4121-9ae8-433fdbae3781/manager/0.log"
Jul 27 20:33:44.598562 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:44.598525    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-dgvzl_f9938b9c-d76a-4c14-b8e1-95dad0295fea/manager/0.log"
Jul 27 20:33:44.632714 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:44.632683    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-f6kfh_12c1ad92-cf64-47f8-aff2-3f1a9aebab7d/kuadrant-console-plugin/0.log"
Jul 27 20:33:44.678226 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:44.678140    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-pvj4x_8abd6d76-25d6-4667-a7d2-051a753e8972/registry-server/0.log"
Jul 27 20:33:44.908507 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:44.907467    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dnrwh_2d7fce82-17ec-4d6c-ae75-c76cdafcd56b/manager/0.log"
Jul 27 20:33:45.032560 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:45.032136    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hx725_6fa4c9e5-3210-49bb-a40a-f14b21d5af13/manager/0.log"
Jul 27 20:33:47.161211 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:47.161175    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-tn62g_527491fe-e2e2-4855-87b7-57377502f9a5/monitoring-plugin/0.log"
Jul 27 20:33:47.381115 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:47.381085    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xlpkj_fa8550e8-f12a-418a-85fb-5a39fe8f8f3c/node-exporter/0.log"
Jul 27 20:33:47.401181 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:47.401108    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xlpkj_fa8550e8-f12a-418a-85fb-5a39fe8f8f3c/kube-rbac-proxy/0.log"
Jul 27 20:33:47.421538 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:47.421425    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xlpkj_fa8550e8-f12a-418a-85fb-5a39fe8f8f3c/init-textfile/0.log"
Jul 27 20:33:47.546430 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:47.546399    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a78bcc2-e19e-45da-946c-5b91c8159602/prometheus/0.log"
Jul 27 20:33:47.567021 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:47.566988    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a78bcc2-e19e-45da-946c-5b91c8159602/config-reloader/0.log"
Jul 27 20:33:47.587779 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:47.587750    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a78bcc2-e19e-45da-946c-5b91c8159602/thanos-sidecar/0.log"
Jul 27 20:33:47.614063 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:47.614039    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a78bcc2-e19e-45da-946c-5b91c8159602/kube-rbac-proxy-web/0.log"
Jul 27 20:33:47.640330 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:47.640300    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a78bcc2-e19e-45da-946c-5b91c8159602/kube-rbac-proxy/0.log"
Jul 27 20:33:47.663914 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:47.663878    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a78bcc2-e19e-45da-946c-5b91c8159602/kube-rbac-proxy-thanos/0.log"
Jul 27 20:33:47.696452 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:47.696359    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a78bcc2-e19e-45da-946c-5b91c8159602/init-config-reloader/0.log"
Jul 27 20:33:47.811331 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:47.811304    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f998b6b8f-jqfgd_a24af1c0-25af-4a5d-9329-d017e80af81b/telemeter-client/0.log"
Jul 27 20:33:47.830268 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:47.830240    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f998b6b8f-jqfgd_a24af1c0-25af-4a5d-9329-d017e80af81b/reload/0.log"
Jul 27 20:33:47.849637 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:47.849611    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f998b6b8f-jqfgd_a24af1c0-25af-4a5d-9329-d017e80af81b/kube-rbac-proxy/0.log"
Jul 27 20:33:49.147210 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.147183    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-hsv74_7475366e-af8a-4306-b809-22f9239e51d7/networking-console-plugin/0.log"
Jul 27 20:33:49.269048 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.269011    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"]
Jul 27 20:33:49.275930 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.275901    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.278695 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.278663    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sgjdf\"/\"default-dockercfg-spjbp\""
Jul 27 20:33:49.280015 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.279619    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"]
Jul 27 20:33:49.378336 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.378292    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-748fx\" (UniqueName: \"kubernetes.io/projected/147c1538-af67-4a51-886b-b31cf977264f-kube-api-access-748fx\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.378719 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.378696    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/147c1538-af67-4a51-886b-b31cf977264f-lib-modules\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.378937 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.378901    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/147c1538-af67-4a51-886b-b31cf977264f-podres\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.379031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.378967    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/147c1538-af67-4a51-886b-b31cf977264f-proc\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.379031 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.379014    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/147c1538-af67-4a51-886b-b31cf977264f-sys\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.480008 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.479916    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/147c1538-af67-4a51-886b-b31cf977264f-lib-modules\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.480008 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.479974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/147c1538-af67-4a51-886b-b31cf977264f-podres\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.480008 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.480004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/147c1538-af67-4a51-886b-b31cf977264f-proc\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.480346 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.480045    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/147c1538-af67-4a51-886b-b31cf977264f-sys\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.480346 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.480126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-748fx\" (UniqueName: \"kubernetes.io/projected/147c1538-af67-4a51-886b-b31cf977264f-kube-api-access-748fx\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.480346 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.480131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/147c1538-af67-4a51-886b-b31cf977264f-podres\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.480346 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.480146    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/147c1538-af67-4a51-886b-b31cf977264f-lib-modules\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.480346 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.480206    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/147c1538-af67-4a51-886b-b31cf977264f-sys\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.480346 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.480262    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/147c1538-af67-4a51-886b-b31cf977264f-proc\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.491550 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.491459    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-748fx\" (UniqueName: \"kubernetes.io/projected/147c1538-af67-4a51-886b-b31cf977264f-kube-api-access-748fx\") pod \"perf-node-gather-daemonset-z4pw4\" (UID: \"147c1538-af67-4a51-886b-b31cf977264f\") " pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.590366 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.590328    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:49.788254 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:49.785717    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"]
Jul 27 20:33:49.789929 ip-10-0-142-140 kubenswrapper[2562]: W0727 20:33:49.789898    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod147c1538_af67_4a51_886b_b31cf977264f.slice/crio-de95654708c28f51b509215df52bc4aa406cdee7499af5c1075d2b20d86291e9 WatchSource:0}: Error finding container de95654708c28f51b509215df52bc4aa406cdee7499af5c1075d2b20d86291e9: Status 404 returned error can't find the container with id de95654708c28f51b509215df52bc4aa406cdee7499af5c1075d2b20d86291e9
Jul 27 20:33:50.334425 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:50.334380    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4" event={"ID":"147c1538-af67-4a51-886b-b31cf977264f","Type":"ContainerStarted","Data":"0c847f2db57d9482b4138693013800e8a59984eba1c0c00662ee6999b4317c64"}
Jul 27 20:33:50.334929 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:50.334452    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4" event={"ID":"147c1538-af67-4a51-886b-b31cf977264f","Type":"ContainerStarted","Data":"de95654708c28f51b509215df52bc4aa406cdee7499af5c1075d2b20d86291e9"}
Jul 27 20:33:50.334929 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:50.334747    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:50.352242 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:50.352178    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4" podStartSLOduration=1.352157595 podStartE2EDuration="1.352157595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:33:50.350083552 +0000 UTC m=+3244.901999972" watchObservedRunningTime="2026-07-27 20:33:50.352157595 +0000 UTC m=+3244.904074013"
Jul 27 20:33:50.400644 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:50.400612    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58b9cbd668-4wfzv_c080bc09-35ce-46a3-9b82-280e9c8e3355/console/0.log"
Jul 27 20:33:50.440007 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:50.439975    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-w6bsx_db3b9846-fa9a-40d0-b2fa-031c0c6212b2/download-server/0.log"
Jul 27 20:33:52.030604 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:52.030575    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9vv5w_765ab079-566d-47ea-b202-f685ec0116e2/dns/0.log"
Jul 27 20:33:52.050266 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:52.050224    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9vv5w_765ab079-566d-47ea-b202-f685ec0116e2/kube-rbac-proxy/0.log"
Jul 27 20:33:52.185154 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:52.185118    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bglxw_00b00751-0c4b-431b-8699-6aad21590c67/dns-node-resolver/0.log"
Jul 27 20:33:52.717314 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:52.717280    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-78f95f545c-r7f6p_930cfab4-26cf-4c8d-84d8-f87825942497/registry/0.log"
Jul 27 20:33:52.739544 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:52.739517    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8qf2p_d50693b0-f760-4b28-8b31-514885f144ab/node-ca/0.log"
Jul 27 20:33:53.704055 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:53.704022    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw_64c987d6-44b0-4d39-8bb4-a4e5cc37e157/istio-proxy/0.log"
Jul 27 20:33:54.026897 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:54.026819    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fd5d97dbc-fllhj_8799eeb6-ba19-46de-85e6-007eb62dcf70/kube-auth-proxy/0.log"
Jul 27 20:33:54.055844 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:54.055813    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vbkhz_861d7f38-16d8-41ed-bf20-d15a7bcc6cbf/istio-proxy/0.log"
Jul 27 20:33:54.095040 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:54.095005    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vbkhz_861d7f38-16d8-41ed-bf20-d15a7bcc6cbf/istio-proxy/1.log"
Jul 27 20:33:54.127658 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:54.127625    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-952m8_6da65f36-1bc2-44a6-b6f7-52cf19b524d6/payload-pre-processing/0.log"
Jul 27 20:33:54.157055 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:54.157023    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wrscr_65bda14a-6ab1-406e-8754-305218a633ab/payload-processing/0.log"
Jul 27 20:33:54.980684 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:54.980650    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pg89k_b38c083a-9531-4819-9439-7a151799aa81/serve-healthcheck-canary/0.log"
Jul 27 20:33:55.495815 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:55.495774    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l46qm_625ce43b-c9a8-4a13-91e3-b6e014b59c1b/kube-rbac-proxy/0.log"
Jul 27 20:33:55.516187 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:55.516153    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l46qm_625ce43b-c9a8-4a13-91e3-b6e014b59c1b/exporter/0.log"
Jul 27 20:33:55.536996 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:55.536965    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l46qm_625ce43b-c9a8-4a13-91e3-b6e014b59c1b/extractor/0.log"
Jul 27 20:33:56.353424 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:56.353394    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sgjdf/perf-node-gather-daemonset-z4pw4"
Jul 27 20:33:57.755737 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:57.755704    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pxkfm_b4cd6b2f-c1db-4267-a123-ba564753c91d/manager/0.log"
Jul 27 20:33:58.038435 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:58.038364    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bwq5w_21a0d151-3240-46fe-8c19-e64e2d60e684/manager/0.log"
Jul 27 20:33:58.363601 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:58.363514    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8b889c85-k7rvz_ab7521f3-a8cb-49a4-a2bd-d3ce27080dc5/manager/0.log"
Jul 27 20:33:58.382493 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:58.382449    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-shdxq_a7ed7694-6b8f-4519-bc40-9ac72f89945a/server/0.log"
Jul 27 20:33:58.500755 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:58.500727    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-p79zv_a3ee9a3c-80b0-4db1-b61f-f34240e5f2aa/manager/0.log"
Jul 27 20:33:58.524443 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:58.524414    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df5588f67-626t9_56f55834-fe4e-403f-a5ad-1d21555238a9/manager/0.log"
Jul 27 20:33:58.544239 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:33:58.544205    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df5588f67-626t9_56f55834-fe4e-403f-a5ad-1d21555238a9/copy-manifests/0.log"
Jul 27 20:34:00.095391 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:00.095364    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw_64c987d6-44b0-4d39-8bb4-a4e5cc37e157/istio-proxy/0.log"
Jul 27 20:34:00.384672 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:00.384638    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fd5d97dbc-fllhj_8799eeb6-ba19-46de-85e6-007eb62dcf70/kube-auth-proxy/0.log"
Jul 27 20:34:00.406951 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:00.406884    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vbkhz_861d7f38-16d8-41ed-bf20-d15a7bcc6cbf/istio-proxy/0.log"
Jul 27 20:34:00.442530 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:00.442499    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vbkhz_861d7f38-16d8-41ed-bf20-d15a7bcc6cbf/istio-proxy/1.log"
Jul 27 20:34:00.465953 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:00.465927    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-952m8_6da65f36-1bc2-44a6-b6f7-52cf19b524d6/payload-pre-processing/0.log"
Jul 27 20:34:00.490297 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:00.490249    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wrscr_65bda14a-6ab1-406e-8754-305218a633ab/payload-processing/0.log"
Jul 27 20:34:01.879549 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:01.879518    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c4lrcw_64c987d6-44b0-4d39-8bb4-a4e5cc37e157/istio-proxy/0.log"
Jul 27 20:34:02.197803 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:02.197719    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fd5d97dbc-fllhj_8799eeb6-ba19-46de-85e6-007eb62dcf70/kube-auth-proxy/0.log"
Jul 27 20:34:02.220012 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:02.219979    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vbkhz_861d7f38-16d8-41ed-bf20-d15a7bcc6cbf/istio-proxy/0.log"
Jul 27 20:34:02.257624 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:02.257578    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vbkhz_861d7f38-16d8-41ed-bf20-d15a7bcc6cbf/istio-proxy/1.log"
Jul 27 20:34:02.288350 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:02.288312    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-952m8_6da65f36-1bc2-44a6-b6f7-52cf19b524d6/payload-pre-processing/0.log"
Jul 27 20:34:02.314918 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:02.314889    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wrscr_65bda14a-6ab1-406e-8754-305218a633ab/payload-processing/0.log"
Jul 27 20:34:04.947930 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:04.947882    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-fh6px_f052f509-6952-4c14-a833-cee46a6f8564/migrator/0.log"
Jul 27 20:34:04.968354 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:04.968322    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-fh6px_f052f509-6952-4c14-a833-cee46a6f8564/graceful-termination/0.log"
Jul 27 20:34:06.277666 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:06.277635    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-42l2f_63bd477d-eac4-4ca6-b5d3-4734ba278992/kube-multus/0.log"
Jul 27 20:34:06.300455 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:06.300423    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mwwb_8413e456-000f-457a-9ad6-a337819268cc/kube-multus-additional-cni-plugins/0.log"
Jul 27 20:34:06.326681 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:06.326652    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mwwb_8413e456-000f-457a-9ad6-a337819268cc/egress-router-binary-copy/0.log"
Jul 27 20:34:06.353833 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:06.353810    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mwwb_8413e456-000f-457a-9ad6-a337819268cc/cni-plugins/0.log"
Jul 27 20:34:06.376030 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:06.376007    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mwwb_8413e456-000f-457a-9ad6-a337819268cc/bond-cni-plugin/0.log"
Jul 27 20:34:06.395933 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:06.395907    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mwwb_8413e456-000f-457a-9ad6-a337819268cc/routeoverride-cni/0.log"
Jul 27 20:34:06.416206 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:06.416180    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mwwb_8413e456-000f-457a-9ad6-a337819268cc/whereabouts-cni-bincopy/0.log"
Jul 27 20:34:06.435034 ip-10-0-142-140 kubenswrapper[2562]: I0727 20:34:06.435013    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7mwwb_8413e456-000f-457a-9ad6-a337819268cc/whereabouts-cni/0.log"