Jul 14 15:21:15.538783 ip-10-0-136-210 systemd[1]: Starting Kubernetes Kubelet...
Jul 14 15:21:15.927810 ip-10-0-136-210 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 15:21:15.927810 ip-10-0-136-210 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 14 15:21:15.927810 ip-10-0-136-210 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 15:21:15.927810 ip-10-0-136-210 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 14 15:21:15.927810 ip-10-0-136-210 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 15:21:15.928783 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.928697    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 14 15:21:15.932999 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.932984    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 15:21:15.933036 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933001    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 15:21:15.933036 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933007    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 15:21:15.933036 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933012    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 15:21:15.933036 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933015    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 15:21:15.933036 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933018    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 15:21:15.933036 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933021    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 15:21:15.933036 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933024    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933090    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933095    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933098    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933101    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933104    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933107    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933113    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933116    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933119    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933121    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933123    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933126    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933128    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933130    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933133    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933135    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933138    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933140    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933143    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 15:21:15.933202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933145    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933147    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933150    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933152    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933154    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933156    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933159    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933161    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933163    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933166    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933168    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933170    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933172    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933175    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933179    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933182    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933184    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933186    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933189    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933191    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 15:21:15.933676 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933194    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933196    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933198    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933201    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933203    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933206    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933208    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933210    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933213    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933216    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933218    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933220    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933223    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933225    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933233    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933235    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933238    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933240    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933243    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933245    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 15:21:15.934204 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933248    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933250    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933253    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933257    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933261    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933265    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933269    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933271    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933274    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933277    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933279    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933283    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933286    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933288    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933291    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933295    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933297    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933299    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933302    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 15:21:15.934689 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933305    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933307    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933310    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933313    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933315    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933317    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933320    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933322    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933324    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933326    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933329    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933331    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.933333    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934020    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934027    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934030    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934033    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934035    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934039    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934042    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 15:21:15.935184 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934045    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934048    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934050    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934053    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934056    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934059    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934061    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934064    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934066    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934069    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934071    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934074    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934077    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934079    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934081    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934084    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934086    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934088    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934091    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934093    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 15:21:15.935717 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934096    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934098    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934101    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934103    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934105    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934108    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934110    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934112    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934115    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934117    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934120    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934124    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934126    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934129    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934144    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934147    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934149    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934152    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934154    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934157    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 15:21:15.936222 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934159    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934162    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934164    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934167    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934169    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934171    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934174    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934176    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934179    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934181    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934184    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934186    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934189    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934193    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934196    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934200    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934203    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934206    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934208    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934211    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 15:21:15.936874 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934213    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934216    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934218    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934223    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934226    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934229    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934232    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934234    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934238    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934241    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934244    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934246    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934249    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934252    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934254    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934257    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934259    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934262    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934264    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 15:21:15.937386 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934266    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934269    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934271    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934274    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934276    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934279    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934281    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934283    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934285    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934288    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934290    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934293    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934295    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934669    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934673    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934676    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934681    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934690    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934693    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 15:21:15.937862 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934696    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934698    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934701    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934704    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934706    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934708    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934711    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934713    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934716    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934719    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934721    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934724    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934726    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934729    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934731    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934733    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934736    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934738    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934740    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934742    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934745    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 15:21:15.938311 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934747    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934750    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934752    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934754    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934757    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934759    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934761    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934764    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934766    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934769    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934772    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934775    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934777    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934780    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934782    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934784    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934787    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934789    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934792    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934794    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 15:21:15.938853 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934796    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934798    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934801    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934803    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934805    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934807    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934810    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934812    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934814    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934823    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934827    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934830    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934833    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934835    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934837    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934843    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934847    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934849    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934871    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 15:21:15.939349 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934875    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934879    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934882    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934885    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934889    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934892    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934895    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934898    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934901    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934903    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934906    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934908    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934911    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934914    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934916    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934919    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934922    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934924    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934927    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934930    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 15:21:15.939814 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934932    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934935    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934937    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934940    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934942    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934945    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934947    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934950    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934952    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934955    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934957    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934960    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.934962    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935037    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935047    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935065    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935071    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935079    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935084    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935096    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935101    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 14 15:21:15.940286 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935105    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935108    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935111    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935114    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935117    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935120    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935122    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935126    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935128    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935131    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935139    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935142    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935144    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935148    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935151    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935154    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935157    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935160    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935163    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935166    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935169    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935172    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935176    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935179    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 14 15:21:15.940777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935183    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935185    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935188    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935191    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935194    2563 flags.go:64] FLAG: --enable-server="true"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935198    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935204    2563 flags.go:64] FLAG: --event-burst="100"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935207    2563 flags.go:64] FLAG: --event-qps="50"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935210    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935213    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935216    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935220    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935222    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935225    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935228    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935231    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935234    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935237    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935244    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935247    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935250    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935253    2563 flags.go:64] FLAG: --feature-gates=""
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935257    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935260    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935263    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 14 15:21:15.941336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935266    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935269    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935272    2563 flags.go:64] FLAG: --help="false"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935274    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-136-210.ec2.internal"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935277    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935280    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935283    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935287    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935290    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935293    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935295    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935298    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935301    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935306    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935309    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935312    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935315    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935317    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935320    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935323    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935326    2563 flags.go:64] FLAG: --lock-file=""
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935328    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935331    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935334    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 14 15:21:15.941989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935339    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935344    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935347    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935349    2563 flags.go:64] FLAG: --logging-format="text"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935352    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935355    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935358    2563 flags.go:64] FLAG: --manifest-url=""
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935361    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935365    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935368    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935372    2563 flags.go:64] FLAG: --max-pods="110"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935375    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935378    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935381    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935384    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935387    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935389    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935392    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935399    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935402    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935405    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935409    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935412    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 14 15:21:15.942603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935418    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935421    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935425    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935427    2563 flags.go:64] FLAG: --port="10250"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935430    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935433    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01c6948c1ad69f43c"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935436    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935439    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935442    2563 flags.go:64] FLAG: --register-node="true"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935445    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935448    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935452    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935455    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935458    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935461    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935464    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935467    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935470    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935472    2563 flags.go:64] FLAG: --runonce="false"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935475    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935478    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935481    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935483    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935486    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935489    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935492    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 14 15:21:15.943181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935495    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935498    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935526    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935529    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935534    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935537    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935540    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935543    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935548    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935551    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935554    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935558    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935561    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935564    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935566    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935569    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935572    2563 flags.go:64] FLAG: --v="2"
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935577    2563 flags.go:64] FLAG: --version="false"
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935581    2563 flags.go:64] FLAG: --vmodule=""
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935585    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.935588    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935711    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935714    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935718    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 15:21:15.943848 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935721    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935729    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935732    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935735    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935738    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935741    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935744    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935746    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935749    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935751    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935754    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935756    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935760    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935763    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935765    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935773    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935776    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935778    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935781    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935783    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 15:21:15.944432 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935786    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935788    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935791    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935793    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935795    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935800    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935802    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935805    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935807    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935810    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935812    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935815    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935817    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935819    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935822    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935825    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935827    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935830    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935832    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935835    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 15:21:15.944923 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935837    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935839    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935842    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935844    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935848    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935851    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935854    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935856    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935859    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935861    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935864    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935866    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935869    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935871    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935874    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935876    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935878    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935882    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935884    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935887    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 15:21:15.945377 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935889    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935892    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935894    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935897    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935899    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935903    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935906    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935909    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935911    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935914    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935916    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935920    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935924    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935927    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935930    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935932    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935936    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935939    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935941    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 15:21:15.945893 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935944    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935959    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935963    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935965    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935968    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935970    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935972    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935974    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935977    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935979    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935983    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935985    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935987    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935991    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935994    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935997    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 15:21:15.946335 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.935999    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.936699    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.943367    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.943382    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.944151    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.944158    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944213    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944218    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944221    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944224    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944226    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944229    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944231    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944234    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944236    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 15:21:15.946739 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944239    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944242    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944244    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944247    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944250    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944252    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944255    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944258    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944262    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944265    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944268    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944270    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944273    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944275    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944278    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944280    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944283    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944285    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944288    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 15:21:15.947125 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944290    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944294    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944297    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944299    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944302    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944304    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944308    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944311    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944313    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944315    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944318    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944320    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944322    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944325    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944327    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944330    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944332    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944334    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944337    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944340    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 15:21:15.947582 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944342    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944345    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944347    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944350    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944352    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944354    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944357    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944360    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944363    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944366    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944369    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944372    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944374    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944378    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944381    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944384    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944387    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944389    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944392    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 15:21:15.948102 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944394    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944396    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944399    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944402    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944404    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944407    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944409    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944411    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944414    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944416    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944418    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944421    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944423    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944425    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944428    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944430    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944432    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944435    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944437    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944440    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 15:21:15.948699 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944443    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944445    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944448    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944450    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944453    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944455    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944458    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944461    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944464    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944466    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944468    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944472    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.944477    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944605    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944611    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 15:21:15.949428 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944613    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944616    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944618    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944620    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944623    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944626    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944628    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944632    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944635    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944638    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944641    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944643    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944646    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944650    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944652    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944656    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944658    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944660    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944663    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 15:21:15.949896 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944665    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944667    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944670    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944672    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944675    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944677    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944680    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944682    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944685    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944688    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944690    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944692    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944695    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944697    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944700    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944702    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944704    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944707    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944709    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944711    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 15:21:15.950405 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944714    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944716    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944718    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944721    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944723    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944726    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944728    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944730    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944733    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944735    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944738    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944740    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944742    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944744    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944747    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944749    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944751    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944754    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944756    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944758    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 15:21:15.951027 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944761    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944764    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944766    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944768    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944771    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944773    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944775    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944777    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944780    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944782    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944784    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944787    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944789    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944791    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944794    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944796    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944798    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944800    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944803    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944805    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 15:21:15.951544 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944807    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944811    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944814    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944817    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944820    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944822    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944824    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944827    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944830    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944832    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944835    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944837    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944839    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944842    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944844    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944847    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944849    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 15:21:15.952075 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:15.944851    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 15:21:15.952467 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.944856    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 15:21:15.952467 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.944968    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 14 15:21:15.952467 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.947954    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 14 15:21:15.952467 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.950430    2563 server.go:1019] "Starting client certificate rotation"
Jul 14 15:21:15.952467 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.950545    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 15:21:15.952467 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.950583    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 15:21:15.954109 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.954092    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 15:21:15.955260 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.955244    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 15:21:15.964070 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.964054    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 14 15:21:15.965972 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.965958    2563 log.go:25] "Validated CRI v1 image API"
Jul 14 15:21:15.966831 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.966808    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 14 15:21:15.972012 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.971994    2563 fs.go:135] Filesystem UUIDs: map[38fc359e-2ef1-412e-920b-6c17dcd7f166:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cf5a6510-f164-4c4d-a514-930bc9820021:/dev/nvme0n1p3]
Jul 14 15:21:15.972077 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.972013    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 14 15:21:15.978141 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.978024    2563 manager.go:216] Machine: {Timestamp:2026-07-14 15:21:15.975993041 +0000 UTC m=+0.317258959 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199329 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d376dfef7c4ba6309a3929b8cbb81 SystemUUID:ec2d376d-fef7-c4ba-6309-a3929b8cbb81 BootID:40e1bdd0-f942-421d-ad21-cda3b73e24aa Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f3:b1:d6:a2:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f3:b1:d6:a2:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:49:9d:23:b7:c2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 14 15:21:15.978141 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.978137    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 14 15:21:15.978271 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.978229    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 14 15:21:15.980268 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.980253    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 15:21:15.980309 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.980272    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 14 15:21:15.980485 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.980311    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-210.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 14 15:21:15.980982 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.980971    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 14 15:21:15.981011 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.980984    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 14 15:21:15.981011 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.980992    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 15:21:15.981610 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.981600    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 15:21:15.982360 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.982349    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 14 15:21:15.985175 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.985164    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 15:21:15.985290 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.985279    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 14 15:21:15.985325 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.985314    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 14 15:21:15.985355 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.985327    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 14 15:21:15.985355 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.985342    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 14 15:21:15.985355 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.985352    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 14 15:21:15.985459 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.985362    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 14 15:21:15.985758 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.985740    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 15:21:15.985758 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.985763    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 15:21:15.987981 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.987962    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 14 15:21:15.991463 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.991436    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 14 15:21:15.991579 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.991549    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 14 15:21:15.992161 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.992149    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 14 15:21:15.992223 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.992166    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 14 15:21:15.992223 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.992172    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 14 15:21:15.992223 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.992178    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 14 15:21:15.992223 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.992185    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 14 15:21:15.992223 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.992190    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 14 15:21:15.992393 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.992361    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 14 15:21:15.992393 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.992368    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 14 15:21:15.992393 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.992375    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 14 15:21:15.992393 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.992381    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 14 15:21:15.992393 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.992389    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 14 15:21:15.992547 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.992396    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 14 15:21:15.992547 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.992425    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 14 15:21:15.992547 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.992430    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 14 15:21:15.995844 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.995831    2563 server.go:1268] "Started kubelet"
Jul 14 15:21:15.995933 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.995892    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 14 15:21:15.996005 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.995929    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 14 15:21:15.996050 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.996032    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 14 15:21:15.996269 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.996249    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-210.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 14 15:21:15.996519 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:15.996481    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 14 15:21:15.996814 ip-10-0-136-210 systemd[1]: Started Kubernetes Kubelet.
Jul 14 15:21:15.996929 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:15.996882    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 14 15:21:15.999635 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.998389    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 14 15:21:15.999635 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:15.998547    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 14 15:21:16.001222 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.000205    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-210.ec2.internal.18c2312dfdf034b2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-210.ec2.internal,UID:ip-10-0-136-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-210.ec2.internal,},FirstTimestamp:2026-07-14 15:21:15.99580485 +0000 UTC m=+0.337070747,LastTimestamp:2026-07-14 15:21:15.99580485 +0000 UTC m=+0.337070747,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-210.ec2.internal,}"
Jul 14 15:21:16.002691 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.002664    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 14 15:21:16.002917 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.002903    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fksfc"
Jul 14 15:21:16.004163 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.004149    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 14 15:21:16.004235 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.004155    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 14 15:21:16.004290 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.004269    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-210.ec2.internal\" not found"
Jul 14 15:21:16.004713 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.004692    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 14 15:21:16.004713 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.004695    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 14 15:21:16.004830 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.004721    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 14 15:21:16.004877 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.004834    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 14 15:21:16.004877 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.004842    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 14 15:21:16.005250 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.005231    2563 factory.go:55] Registering systemd factory
Jul 14 15:21:16.005250 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.005252    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 14 15:21:16.005466 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.005446    2563 factory.go:153] Registering CRI-O factory
Jul 14 15:21:16.005466 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.005462    2563 factory.go:223] Registration of the crio container factory successfully
Jul 14 15:21:16.005571 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.005533    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 14 15:21:16.005571 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.005555    2563 factory.go:103] Registering Raw factory
Jul 14 15:21:16.005571 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.005570    2563 manager.go:1196] Started watching for new ooms in manager
Jul 14 15:21:16.005961 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.005949    2563 manager.go:349] Starting recovery of all containers
Jul 14 15:21:16.006329 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.006309    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 14 15:21:16.006406 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.006389    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 14 15:21:16.013231 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.013213    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fksfc"
Jul 14 15:21:16.017541 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.017380    2563 manager.go:354] Recovery completed
Jul 14 15:21:16.021987 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.021968    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 15:21:16.024319 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.024302    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 15:21:16.024402 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.024333    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 15:21:16.024402 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.024343    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasSufficientPID"
Jul 14 15:21:16.024975 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.024948    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 14 15:21:16.024975 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.024966    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 14 15:21:16.025087 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.024983    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 15:21:16.025087 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.024952    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 15:21:16.025800 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.025784    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 15:21:16.025869 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.025816    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 15:21:16.025869 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.025830    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasSufficientPID"
Jul 14 15:21:16.026136 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.026124    2563 policy_none.go:49] "None policy: Start"
Jul 14 15:21:16.026584 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.026568    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 14 15:21:16.026647 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.026594    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 14 15:21:16.027596 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.027585    2563 policy_none.go:47] "Start"
Jul 14 15:21:16.060064 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.060047    2563 manager.go:337] "Starting Device Plugin manager"
Jul 14 15:21:16.060184 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.060086    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 14 15:21:16.060184 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.060099    2563 server.go:87] "Starting device plugin registration server"
Jul 14 15:21:16.060376 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.060364    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 14 15:21:16.060419 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.060378    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 14 15:21:16.060518 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.060487    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 14 15:21:16.060578 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.060567    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 14 15:21:16.060615 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.060578    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 14 15:21:16.061047 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.061023    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 14 15:21:16.061155 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.061075    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-210.ec2.internal\" not found"
Jul 14 15:21:16.160992 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.160964    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 15:21:16.162074 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.161924    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 15:21:16.162074 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.161956    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 15:21:16.162074 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.161965    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasSufficientPID"
Jul 14 15:21:16.162074 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.161993    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.170690 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.170672    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.170734 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.170695    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-210.ec2.internal\": node \"ip-10-0-136-210.ec2.internal\" not found"
Jul 14 15:21:16.188329 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.188261    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-210.ec2.internal\" not found"
Jul 14 15:21:16.208207 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.208183    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 14 15:21:16.209534 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.209498    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 14 15:21:16.209602 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.209561    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 14 15:21:16.209602 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.209587    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 14 15:21:16.209686 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.209625    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 14 15:21:16.211235 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.211220    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 15:21:16.288965 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.288923    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-210.ec2.internal\" not found"
Jul 14 15:21:16.310336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.310313    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-210.ec2.internal"]
Jul 14 15:21:16.310396 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.310376    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 15:21:16.311346 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.311330    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 15:21:16.311433 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.311359    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 15:21:16.311433 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.311369    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasSufficientPID"
Jul 14 15:21:16.312645 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.312633    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 15:21:16.312774 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.312762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.312808 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.312791    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 15:21:16.313335 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.313320    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 15:21:16.313415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.313350    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 15:21:16.313415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.313360    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasSufficientPID"
Jul 14 15:21:16.313415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.313320    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 15:21:16.313525 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.313419    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 15:21:16.313525 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.313432    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasSufficientPID"
Jul 14 15:21:16.314710 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.314696    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.314758 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.314721    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 15:21:16.316717 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.316699    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 15:21:16.316812 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.316733    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 15:21:16.316812 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.316748    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeHasSufficientPID"
Jul 14 15:21:16.332041 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.332021    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-210.ec2.internal\" not found" node="ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.336053 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.336039    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-210.ec2.internal\" not found" node="ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.389794 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.389770    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-210.ec2.internal\" not found"
Jul 14 15:21:16.408256 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.408232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a95379fcdfcd10b1894525c55897d3ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal\" (UID: \"a95379fcdfcd10b1894525c55897d3ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.408342 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.408260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a95379fcdfcd10b1894525c55897d3ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal\" (UID: \"a95379fcdfcd10b1894525c55897d3ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.408342 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.408275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d95b182bc9232c53452fc18d92789eb-config\") pod \"kube-apiserver-proxy-ip-10-0-136-210.ec2.internal\" (UID: \"3d95b182bc9232c53452fc18d92789eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.490122 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.490057    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-210.ec2.internal\" not found"
Jul 14 15:21:16.509429 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.509362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a95379fcdfcd10b1894525c55897d3ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal\" (UID: \"a95379fcdfcd10b1894525c55897d3ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.509498 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.509454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a95379fcdfcd10b1894525c55897d3ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal\" (UID: \"a95379fcdfcd10b1894525c55897d3ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.509498 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.509486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d95b182bc9232c53452fc18d92789eb-config\") pod \"kube-apiserver-proxy-ip-10-0-136-210.ec2.internal\" (UID: \"3d95b182bc9232c53452fc18d92789eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.509498 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.509410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a95379fcdfcd10b1894525c55897d3ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal\" (UID: \"a95379fcdfcd10b1894525c55897d3ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.509623 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.509546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a95379fcdfcd10b1894525c55897d3ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal\" (UID: \"a95379fcdfcd10b1894525c55897d3ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.509623 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.509546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d95b182bc9232c53452fc18d92789eb-config\") pod \"kube-apiserver-proxy-ip-10-0-136-210.ec2.internal\" (UID: \"3d95b182bc9232c53452fc18d92789eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.590578 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.590530    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-210.ec2.internal\" not found"
Jul 14 15:21:16.634040 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.634008    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.639150 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.639133    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:16.691603 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.691570    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-210.ec2.internal\" not found"
Jul 14 15:21:16.792133 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.792062    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-210.ec2.internal\" not found"
Jul 14 15:21:16.857430 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.857402    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 15:21:16.892923 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.892900    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-210.ec2.internal\" not found"
Jul 14 15:21:16.949410 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.949378    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 14 15:21:16.949987 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.949545    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 15:21:16.949987 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:16.949560    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 15:21:16.993942 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:16.993919    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-210.ec2.internal\" not found"
Jul 14 15:21:17.018496 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:17.018458    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 15:16:16 +0000 UTC" deadline="2028-02-09 16:03:37.362214939 +0000 UTC"
Jul 14 15:21:17.018496 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:17.018489    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13800h42m20.343730181s"
Jul 14 15:21:17.080750 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:17.080713    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 15:21:17.104726 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:17.104700    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:17.115923 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:17.115902    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 15:21:17.117998 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:17.117976    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal"
Jul 14 15:21:17.128998 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:17.128978    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 15:21:17.154912 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:17.154880    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95379fcdfcd10b1894525c55897d3ca.slice/crio-b21991693a6d9325427cda64c05b4305d1c2d8795c6f40d85374898ae06cc980 WatchSource:0}: Error finding container b21991693a6d9325427cda64c05b4305d1c2d8795c6f40d85374898ae06cc980: Status 404 returned error can't find the container with id b21991693a6d9325427cda64c05b4305d1c2d8795c6f40d85374898ae06cc980
Jul 14 15:21:17.160515 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:17.160498    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 15:21:17.213634 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:17.213579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal" event={"ID":"a95379fcdfcd10b1894525c55897d3ca","Type":"ContainerStarted","Data":"b21991693a6d9325427cda64c05b4305d1c2d8795c6f40d85374898ae06cc980"}
Jul 14 15:21:17.573425 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:17.573401    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 15:21:17.986857 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:17.986767    2563 apiserver.go:52] "Watching apiserver"
Jul 14 15:21:17.999872 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:17.999841    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 14 15:21:18.000313 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.000282    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-tjprj","openshift-dns/node-resolver-9f2sc","openshift-image-registry/node-ca-7phx7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal","openshift-multus/multus-additional-cni-plugins-bxpqz","openshift-network-operator/iptables-alerter-bwj9q","kube-system/konnectivity-agent-rtlt8","kube-system/kube-apiserver-proxy-ip-10-0-136-210.ec2.internal","openshift-multus/multus-l2dj2","openshift-multus/network-metrics-daemon-frtx9","openshift-network-diagnostics/network-check-target-w82g9","openshift-ovn-kubernetes/ovnkube-node-jfnl2","kube-system/global-pull-secret-syncer-w8dm9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"]
Jul 14 15:21:18.002362 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.002337    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rtlt8"
Jul 14 15:21:18.003587 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.003558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:18.003704 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.003657    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:18.005410 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.004876    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 14 15:21:18.005410 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.005042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7phx7"
Jul 14 15:21:18.005879 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.005741    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 14 15:21:18.005879 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.005757    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 14 15:21:18.005879 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.005779    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ncmmm\""
Jul 14 15:21:18.006074 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.005881    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 15:21:18.006343 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.006323    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.007746 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.007727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.008093 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.007841    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pqq4s\""
Jul 14 15:21:18.008093 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.007948    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 14 15:21:18.008233 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.008181    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 14 15:21:18.008233 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.008214    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 14 15:21:18.009529 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.009400    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 14 15:21:18.009529 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.009427    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 14 15:21:18.009529 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.009463    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 14 15:21:18.009529 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.009475    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4qnlv\""
Jul 14 15:21:18.009775 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.009679    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 14 15:21:18.009775 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.009682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bwj9q"
Jul 14 15:21:18.010904 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.010744    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 14 15:21:18.011011 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.010988    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 14 15:21:18.011154 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.011062    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8q5bq\""
Jul 14 15:21:18.012002 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.011984    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.012723 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.012698    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 14 15:21:18.013329 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.013146    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xlsmx\""
Jul 14 15:21:18.013329 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.013213    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 14 15:21:18.013564 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.013539    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:18.013658 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.013632    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:18.014318 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.014299    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 14 15:21:18.014788 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.014767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9f2sc"
Jul 14 15:21:18.014884 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.014865    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 14 15:21:18.015236 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.015209    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jm4ms\""
Jul 14 15:21:18.015316 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.015214    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 14 15:21:18.016325 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.016307    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.017694 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.017674    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 14 15:21:18.017785 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.017693    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 14 15:21:18.017785 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.017730    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9jmmp\""
Jul 14 15:21:18.017967 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.017948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:18.018062 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.018036    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:18.018424 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.018403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2af3c99-9843-496a-823b-f8a97a1dcd9d-host\") pod \"node-ca-7phx7\" (UID: \"d2af3c99-9843-496a-823b-f8a97a1dcd9d\") " pod="openshift-image-registry/node-ca-7phx7"
Jul 14 15:21:18.018532 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.018437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7qqm\" (UniqueName: \"kubernetes.io/projected/d2af3c99-9843-496a-823b-f8a97a1dcd9d-kube-api-access-h7qqm\") pod \"node-ca-7phx7\" (UID: \"d2af3c99-9843-496a-823b-f8a97a1dcd9d\") " pod="openshift-image-registry/node-ca-7phx7"
Jul 14 15:21:18.018532 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.018461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2af3c99-9843-496a-823b-f8a97a1dcd9d-serviceca\") pod \"node-ca-7phx7\" (UID: \"d2af3c99-9843-496a-823b-f8a97a1dcd9d\") " pod="openshift-image-registry/node-ca-7phx7"
Jul 14 15:21:18.018532 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.018477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15614b5c-a711-470b-92b6-7ad03154f6be-agent-certs\") pod \"konnectivity-agent-rtlt8\" (UID: \"15614b5c-a711-470b-92b6-7ad03154f6be\") " pod="kube-system/konnectivity-agent-rtlt8"
Jul 14 15:21:18.018532 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.018497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15614b5c-a711-470b-92b6-7ad03154f6be-konnectivity-ca\") pod \"konnectivity-agent-rtlt8\" (UID: \"15614b5c-a711-470b-92b6-7ad03154f6be\") " pod="kube-system/konnectivity-agent-rtlt8"
Jul 14 15:21:18.018532 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.018528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqm7w\" (UniqueName: \"kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w\") pod \"network-check-target-w82g9\" (UID: \"bf1192e2-ad6e-4efc-964e-3c31e509c8e6\") " pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:18.018846 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.018828    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 14 15:21:18.019226 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.019206    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.019898 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.019869    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 14 15:21:18.019898 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.019888    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 14 15:21:18.020086 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.019940    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nf545\""
Jul 14 15:21:18.020518 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.020499    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 14 15:21:18.021780 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.021763    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 14 15:21:18.022131 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.022110    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 14 15:21:18.022285 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.022265    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 14 15:21:18.022610 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.022591    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mplrs\""
Jul 14 15:21:18.022610 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.022605    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 14 15:21:18.022831 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.022815    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 14 15:21:18.025108 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.024834    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 15:21:18.042260 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.042237    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ljg5v"
Jul 14 15:21:18.052649 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.052603    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ljg5v"
Jul 14 15:21:18.105599 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.105547    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 14 15:21:18.118726 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.118691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:18.118887 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.118743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.118887 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.118799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-multus-cni-dir\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.118887 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.118873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93ac7979-c05b-4b5c-8277-d9098a69b8b5-cni-binary-copy\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.119110 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.118918    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-etc-selinux\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.119110 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.118944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5-host-slash\") pod \"iptables-alerter-bwj9q\" (UID: \"46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5\") " pod="openshift-network-operator/iptables-alerter-bwj9q"
Jul 14 15:21:18.119110 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.118970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2blmg\" (UniqueName: \"kubernetes.io/projected/b9623835-e777-4e7d-95be-70ce05205902-kube-api-access-2blmg\") pod \"node-resolver-9f2sc\" (UID: \"b9623835-e777-4e7d-95be-70ce05205902\") " pod="openshift-dns/node-resolver-9f2sc"
Jul 14 15:21:18.119110 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.118994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-run-k8s-cni-cncf-io\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.119110 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-sysctl-conf\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.119110 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-lib-modules\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.119110 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-var-lib-cni-bin\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.119440 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2af3c99-9843-496a-823b-f8a97a1dcd9d-serviceca\") pod \"node-ca-7phx7\" (UID: \"d2af3c99-9843-496a-823b-f8a97a1dcd9d\") " pod="openshift-image-registry/node-ca-7phx7"
Jul 14 15:21:18.119440 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-sys-fs\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.119440 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx4qb\" (UniqueName: \"kubernetes.io/projected/544bf2b6-164b-4c05-8255-016265e27df5-kube-api-access-rx4qb\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.119440 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-kubelet\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.119440 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-var-lib-cni-multus\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.119440 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-multus-conf-dir\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.119440 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-etc-kubernetes\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.119440 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-sysconfig\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.119440 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-kubernetes\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.119440 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-systemd-units\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.119440 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e71167e-01b8-42b0-9403-8ae910ec77bf-ovnkube-config\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-multus-socket-dir-parent\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119560    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs8h5\" (UniqueName: \"kubernetes.io/projected/93ac7979-c05b-4b5c-8277-d9098a69b8b5-kube-api-access-rs8h5\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7qqm\" (UniqueName: \"kubernetes.io/projected/d2af3c99-9843-496a-823b-f8a97a1dcd9d-kube-api-access-h7qqm\") pod \"node-ca-7phx7\" (UID: \"d2af3c99-9843-496a-823b-f8a97a1dcd9d\") " pod="openshift-image-registry/node-ca-7phx7"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5-iptables-alerter-script\") pod \"iptables-alerter-bwj9q\" (UID: \"46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5\") " pod="openshift-network-operator/iptables-alerter-bwj9q"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-os-release\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-sysctl-d\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-var-lib-kubelet\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119730    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2af3c99-9843-496a-823b-f8a97a1dcd9d-serviceca\") pod \"node-ca-7phx7\" (UID: \"d2af3c99-9843-496a-823b-f8a97a1dcd9d\") " pod="openshift-image-registry/node-ca-7phx7"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e0e4d6e-5859-4df9-9f22-d5e18660a104-tmp\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119766    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-log-socket\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-os-release\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3ae120a-8040-483a-8da6-090ef1e80265-kubelet-config\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-slash\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-run-systemd\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.119914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e71167e-01b8-42b0-9403-8ae910ec77bf-env-overrides\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2af3c99-9843-496a-823b-f8a97a1dcd9d-host\") pod \"node-ca-7phx7\" (UID: \"d2af3c99-9843-496a-823b-f8a97a1dcd9d\") " pod="openshift-image-registry/node-ca-7phx7"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-var-lib-kubelet\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2af3c99-9843-496a-823b-f8a97a1dcd9d-host\") pod \"node-ca-7phx7\" (UID: \"d2af3c99-9843-496a-823b-f8a97a1dcd9d\") " pod="openshift-image-registry/node-ca-7phx7"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.119999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93ac7979-c05b-4b5c-8277-d9098a69b8b5-multus-daemon-config\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fblp\" (UniqueName: \"kubernetes.io/projected/b42135f6-4895-4ca1-8086-1ae81b8006c8-kube-api-access-4fblp\") pod \"network-metrics-daemon-frtx9\" (UID: \"b42135f6-4895-4ca1-8086-1ae81b8006c8\") " pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-run-netns\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-run-ovn\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-systemd\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15614b5c-a711-470b-92b6-7ad03154f6be-konnectivity-ca\") pod \"konnectivity-agent-rtlt8\" (UID: \"15614b5c-a711-470b-92b6-7ad03154f6be\") " pod="kube-system/konnectivity-agent-rtlt8"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-registration-dir\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-run-multus-certs\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-var-lib-openvswitch\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-etc-openvswitch\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-node-log\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.120553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-cni-bin\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-system-cni-dir\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-tuned\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gxjg\" (UniqueName: \"kubernetes.io/projected/0e0e4d6e-5859-4df9-9f22-d5e18660a104-kube-api-access-6gxjg\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e71167e-01b8-42b0-9403-8ae910ec77bf-ovn-node-metrics-cert\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120570    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-system-cni-dir\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-cnibin\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15614b5c-a711-470b-92b6-7ad03154f6be-agent-certs\") pod \"konnectivity-agent-rtlt8\" (UID: \"15614b5c-a711-470b-92b6-7ad03154f6be\") " pod="kube-system/konnectivity-agent-rtlt8"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-socket-dir\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-device-dir\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9623835-e777-4e7d-95be-70ce05205902-hosts-file\") pod \"node-resolver-9f2sc\" (UID: \"b9623835-e777-4e7d-95be-70ce05205902\") " pod="openshift-dns/node-resolver-9f2sc"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs\") pod \"network-metrics-daemon-frtx9\" (UID: \"b42135f6-4895-4ca1-8086-1ae81b8006c8\") " pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9623835-e777-4e7d-95be-70ce05205902-tmp-dir\") pod \"node-resolver-9f2sc\" (UID: \"b9623835-e777-4e7d-95be-70ce05205902\") " pod="openshift-dns/node-resolver-9f2sc"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-run-netns\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-run\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.121225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120840    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15614b5c-a711-470b-92b6-7ad03154f6be-konnectivity-ca\") pod \"konnectivity-agent-rtlt8\" (UID: \"15614b5c-a711-470b-92b6-7ad03154f6be\") " pod="kube-system/konnectivity-agent-rtlt8"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-cni-netd\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3ae120a-8040-483a-8da6-090ef1e80265-dbus\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v52x8\" (UniqueName: \"kubernetes.io/projected/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-kube-api-access-v52x8\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-sys\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-host\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.120987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-run-openvswitch\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.121006    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.121032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzc95\" (UniqueName: \"kubernetes.io/projected/46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5-kube-api-access-dzc95\") pod \"iptables-alerter-bwj9q\" (UID: \"46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5\") " pod="openshift-network-operator/iptables-alerter-bwj9q"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.121051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnwhz\" (UniqueName: \"kubernetes.io/projected/7e71167e-01b8-42b0-9403-8ae910ec77bf-kube-api-access-bnwhz\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.121076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-modprobe-d\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.121106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqm7w\" (UniqueName: \"kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w\") pod \"network-check-target-w82g9\" (UID: \"bf1192e2-ad6e-4efc-964e-3c31e509c8e6\") " pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.121128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.121143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.121158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-cnibin\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.121180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-hostroot\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.121874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.121205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e71167e-01b8-42b0-9403-8ae910ec77bf-ovnkube-script-lib\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.125422 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.125392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15614b5c-a711-470b-92b6-7ad03154f6be-agent-certs\") pod \"konnectivity-agent-rtlt8\" (UID: \"15614b5c-a711-470b-92b6-7ad03154f6be\") " pod="kube-system/konnectivity-agent-rtlt8"
Jul 14 15:21:18.127944 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.127914    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 15:21:18.127944 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.127942    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 15:21:18.128082 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.127956    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bqm7w for pod openshift-network-diagnostics/network-check-target-w82g9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 15:21:18.128082 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.128032    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w podName:bf1192e2-ad6e-4efc-964e-3c31e509c8e6 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:18.628015454 +0000 UTC m=+2.969281340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bqm7w" (UniqueName: "kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w") pod "network-check-target-w82g9" (UID: "bf1192e2-ad6e-4efc-964e-3c31e509c8e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 15:21:18.129273 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.129251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7qqm\" (UniqueName: \"kubernetes.io/projected/d2af3c99-9843-496a-823b-f8a97a1dcd9d-kube-api-access-h7qqm\") pod \"node-ca-7phx7\" (UID: \"d2af3c99-9843-496a-823b-f8a97a1dcd9d\") " pod="openshift-image-registry/node-ca-7phx7"
Jul 14 15:21:18.221707 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-sys-fs\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.221707 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx4qb\" (UniqueName: \"kubernetes.io/projected/544bf2b6-164b-4c05-8255-016265e27df5-kube-api-access-rx4qb\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.221923 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-kubelet\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.221923 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-var-lib-cni-multus\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.221923 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-multus-conf-dir\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.221923 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-etc-kubernetes\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.221923 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-sys-fs\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.221923 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-sysconfig\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.221923 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-kubernetes\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.221923 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-sysconfig\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.221923 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-systemd-units\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.221923 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-kubernetes\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.221923 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-systemd-units\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.222392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.222392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-etc-kubernetes\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.222392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.221977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e71167e-01b8-42b0-9403-8ae910ec77bf-ovnkube-config\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.222392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222002    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-kubelet\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.222392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-multus-socket-dir-parent\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.222392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs8h5\" (UniqueName: \"kubernetes.io/projected/93ac7979-c05b-4b5c-8277-d9098a69b8b5-kube-api-access-rs8h5\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.222392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-multus-socket-dir-parent\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.222392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-var-lib-cni-multus\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.222392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-multus-conf-dir\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.222392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5-iptables-alerter-script\") pod \"iptables-alerter-bwj9q\" (UID: \"46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5\") " pod="openshift-network-operator/iptables-alerter-bwj9q"
Jul 14 15:21:18.222392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-os-release\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.222392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-sysctl-d\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.222392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-var-lib-kubelet\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.222392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e0e4d6e-5859-4df9-9f22-d5e18660a104-tmp\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-log-socket\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-os-release\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3ae120a-8040-483a-8da6-090ef1e80265-kubelet-config\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-slash\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-run-systemd\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e71167e-01b8-42b0-9403-8ae910ec77bf-env-overrides\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-var-lib-kubelet\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93ac7979-c05b-4b5c-8277-d9098a69b8b5-multus-daemon-config\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fblp\" (UniqueName: \"kubernetes.io/projected/b42135f6-4895-4ca1-8086-1ae81b8006c8-kube-api-access-4fblp\") pod \"network-metrics-daemon-frtx9\" (UID: \"b42135f6-4895-4ca1-8086-1ae81b8006c8\") " pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e71167e-01b8-42b0-9403-8ae910ec77bf-ovnkube-config\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-run-netns\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-os-release\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-run-netns\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222840    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-slash\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.223045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-run-systemd\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-run-ovn\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-systemd\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.222999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-registration-dir\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-run-multus-certs\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-var-lib-openvswitch\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-etc-openvswitch\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-node-log\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-cni-bin\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-system-cni-dir\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-tuned\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gxjg\" (UniqueName: \"kubernetes.io/projected/0e0e4d6e-5859-4df9-9f22-d5e18660a104-kube-api-access-6gxjg\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e71167e-01b8-42b0-9403-8ae910ec77bf-ovn-node-metrics-cert\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223211    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5-iptables-alerter-script\") pod \"iptables-alerter-bwj9q\" (UID: \"46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5\") " pod="openshift-network-operator/iptables-alerter-bwj9q"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-system-cni-dir\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-cnibin\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.223810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e71167e-01b8-42b0-9403-8ae910ec77bf-env-overrides\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-socket-dir\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-var-lib-kubelet\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-device-dir\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9623835-e777-4e7d-95be-70ce05205902-hosts-file\") pod \"node-resolver-9f2sc\" (UID: \"b9623835-e777-4e7d-95be-70ce05205902\") " pod="openshift-dns/node-resolver-9f2sc"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-var-lib-openvswitch\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs\") pod \"network-metrics-daemon-frtx9\" (UID: \"b42135f6-4895-4ca1-8086-1ae81b8006c8\") " pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9623835-e777-4e7d-95be-70ce05205902-tmp-dir\") pod \"node-resolver-9f2sc\" (UID: \"b9623835-e777-4e7d-95be-70ce05205902\") " pod="openshift-dns/node-resolver-9f2sc"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-run-netns\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-run\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-cni-netd\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3ae120a-8040-483a-8da6-090ef1e80265-dbus\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-sysctl-d\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223517    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v52x8\" (UniqueName: \"kubernetes.io/projected/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-kube-api-access-v52x8\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-sys\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223544    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-var-lib-kubelet\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.224507 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-host\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-run-openvswitch\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzc95\" (UniqueName: \"kubernetes.io/projected/46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5-kube-api-access-dzc95\") pod \"iptables-alerter-bwj9q\" (UID: \"46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5\") " pod="openshift-network-operator/iptables-alerter-bwj9q"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnwhz\" (UniqueName: \"kubernetes.io/projected/7e71167e-01b8-42b0-9403-8ae910ec77bf-kube-api-access-bnwhz\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-modprobe-d\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93ac7979-c05b-4b5c-8277-d9098a69b8b5-multus-daemon-config\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-cnibin\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-hostroot\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e71167e-01b8-42b0-9403-8ae910ec77bf-ovnkube-script-lib\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-multus-cni-dir\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93ac7979-c05b-4b5c-8277-d9098a69b8b5-cni-binary-copy\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.223982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-etc-selinux\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.225324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5-host-slash\") pod \"iptables-alerter-bwj9q\" (UID: \"46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5\") " pod="openshift-network-operator/iptables-alerter-bwj9q"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2blmg\" (UniqueName: \"kubernetes.io/projected/b9623835-e777-4e7d-95be-70ce05205902-kube-api-access-2blmg\") pod \"node-resolver-9f2sc\" (UID: \"b9623835-e777-4e7d-95be-70ce05205902\") " pod="openshift-dns/node-resolver-9f2sc"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-run-k8s-cni-cncf-io\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-sysctl-conf\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-lib-modules\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-var-lib-cni-bin\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-var-lib-cni-bin\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224294    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-sys\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-host\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-node-log\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-etc-openvswitch\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-system-cni-dir\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-run-openvswitch\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-os-release\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-log-socket\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3ae120a-8040-483a-8da6-090ef1e80265-kubelet-config\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-run-ovn\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.226415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-systemd\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-registration-dir\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-modprobe-d\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.224885    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-run-multus-certs\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.225013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9623835-e777-4e7d-95be-70ce05205902-hosts-file\") pod \"node-resolver-9f2sc\" (UID: \"b9623835-e777-4e7d-95be-70ce05205902\") " pod="openshift-dns/node-resolver-9f2sc"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.225300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-run-netns\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.225387    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.225418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.225500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-run\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.225505    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-cnibin\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.225556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-hostroot\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.226074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e71167e-01b8-42b0-9403-8ae910ec77bf-ovnkube-script-lib\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.226106    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.226170    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.226171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-cni-netd\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.226315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3ae120a-8040-483a-8da6-090ef1e80265-dbus\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.226329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.226397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-multus-cni-dir\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.226818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.227351 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.226876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93ac7979-c05b-4b5c-8277-d9098a69b8b5-cni-binary-copy\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.226897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-system-cni-dir\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.226947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-cnibin\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.226991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-etc-selinux\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.227086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-sysctl-conf\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.227112    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e0e4d6e-5859-4df9-9f22-d5e18660a104-lib-modules\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.227144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5-host-slash\") pod \"iptables-alerter-bwj9q\" (UID: \"46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5\") " pod="openshift-network-operator/iptables-alerter-bwj9q"
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.227169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e71167e-01b8-42b0-9403-8ae910ec77bf-host-cni-bin\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.227279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-socket-dir\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.227366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93ac7979-c05b-4b5c-8277-d9098a69b8b5-host-run-k8s-cni-cncf-io\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.227424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/544bf2b6-164b-4c05-8255-016265e27df5-device-dir\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.227510    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9623835-e777-4e7d-95be-70ce05205902-tmp-dir\") pod \"node-resolver-9f2sc\" (UID: \"b9623835-e777-4e7d-95be-70ce05205902\") " pod="openshift-dns/node-resolver-9f2sc"
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.227534    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs podName:b42135f6-4895-4ca1-8086-1ae81b8006c8 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:18.727489207 +0000 UTC m=+3.068755092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs") pod "network-metrics-daemon-frtx9" (UID: "b42135f6-4895-4ca1-8086-1ae81b8006c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.227640    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret podName:d3ae120a-8040-483a-8da6-090ef1e80265 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:18.727573194 +0000 UTC m=+3.068839098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret") pod "global-pull-secret-syncer-w8dm9" (UID: "d3ae120a-8040-483a-8da6-090ef1e80265") : object "kube-system"/"original-pull-secret" not registered
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.227729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e0e4d6e-5859-4df9-9f22-d5e18660a104-tmp\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.228186 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.227796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0e0e4d6e-5859-4df9-9f22-d5e18660a104-etc-tuned\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.229753 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.229691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e71167e-01b8-42b0-9403-8ae910ec77bf-ovn-node-metrics-cert\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.232950 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.232911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs8h5\" (UniqueName: \"kubernetes.io/projected/93ac7979-c05b-4b5c-8277-d9098a69b8b5-kube-api-access-rs8h5\") pod \"multus-l2dj2\" (UID: \"93ac7979-c05b-4b5c-8277-d9098a69b8b5\") " pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.233313 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.233287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx4qb\" (UniqueName: \"kubernetes.io/projected/544bf2b6-164b-4c05-8255-016265e27df5-kube-api-access-rx4qb\") pod \"aws-ebs-csi-driver-node-m96ln\" (UID: \"544bf2b6-164b-4c05-8255-016265e27df5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.234276 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.234253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fblp\" (UniqueName: \"kubernetes.io/projected/b42135f6-4895-4ca1-8086-1ae81b8006c8-kube-api-access-4fblp\") pod \"network-metrics-daemon-frtx9\" (UID: \"b42135f6-4895-4ca1-8086-1ae81b8006c8\") " pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:18.235213 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.235194    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnwhz\" (UniqueName: \"kubernetes.io/projected/7e71167e-01b8-42b0-9403-8ae910ec77bf-kube-api-access-bnwhz\") pod \"ovnkube-node-jfnl2\" (UID: \"7e71167e-01b8-42b0-9403-8ae910ec77bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.235928 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.235901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v52x8\" (UniqueName: \"kubernetes.io/projected/5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7-kube-api-access-v52x8\") pod \"multus-additional-cni-plugins-bxpqz\" (UID: \"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7\") " pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.237122 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.237065    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gxjg\" (UniqueName: \"kubernetes.io/projected/0e0e4d6e-5859-4df9-9f22-d5e18660a104-kube-api-access-6gxjg\") pod \"tuned-tjprj\" (UID: \"0e0e4d6e-5859-4df9-9f22-d5e18660a104\") " pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.237212 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.237185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzc95\" (UniqueName: \"kubernetes.io/projected/46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5-kube-api-access-dzc95\") pod \"iptables-alerter-bwj9q\" (UID: \"46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5\") " pod="openshift-network-operator/iptables-alerter-bwj9q"
Jul 14 15:21:18.238936 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.238911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2blmg\" (UniqueName: \"kubernetes.io/projected/b9623835-e777-4e7d-95be-70ce05205902-kube-api-access-2blmg\") pod \"node-resolver-9f2sc\" (UID: \"b9623835-e777-4e7d-95be-70ce05205902\") " pod="openshift-dns/node-resolver-9f2sc"
Jul 14 15:21:18.315048 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.314990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rtlt8"
Jul 14 15:21:18.322740 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.322559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7phx7"
Jul 14 15:21:18.330670 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.330646    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l2dj2"
Jul 14 15:21:18.334575 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.334549    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bxpqz"
Jul 14 15:21:18.341423 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.341396    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bwj9q"
Jul 14 15:21:18.354652 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.354625    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tjprj"
Jul 14 15:21:18.361659 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.361639    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9f2sc"
Jul 14 15:21:18.370230 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.369849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:18.375712 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.375691    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln"
Jul 14 15:21:18.395555 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:18.395524    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9623835_e777_4e7d_95be_70ce05205902.slice/crio-7e269dc979bda1bafe680be4f99105a03a85f57a862cd9dec2341ced7a2368c4 WatchSource:0}: Error finding container 7e269dc979bda1bafe680be4f99105a03a85f57a862cd9dec2341ced7a2368c4: Status 404 returned error can't find the container with id 7e269dc979bda1bafe680be4f99105a03a85f57a862cd9dec2341ced7a2368c4
Jul 14 15:21:18.397126 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:18.397104    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ac7979_c05b_4b5c_8277_d9098a69b8b5.slice/crio-7b6703ec5e61f4f3ea2770eee506b850081a7bdde93ac2fe2d612c36b678021e WatchSource:0}: Error finding container 7b6703ec5e61f4f3ea2770eee506b850081a7bdde93ac2fe2d612c36b678021e: Status 404 returned error can't find the container with id 7b6703ec5e61f4f3ea2770eee506b850081a7bdde93ac2fe2d612c36b678021e
Jul 14 15:21:18.400757 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:18.400718    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fecf6a4_135a_41cc_b0d7_cdd720a9d6f7.slice/crio-f7b57c18abbf12a8c93b161ad7fa52ea75438cbbb5c6f1ca778085e55e8aaa49 WatchSource:0}: Error finding container f7b57c18abbf12a8c93b161ad7fa52ea75438cbbb5c6f1ca778085e55e8aaa49: Status 404 returned error can't find the container with id f7b57c18abbf12a8c93b161ad7fa52ea75438cbbb5c6f1ca778085e55e8aaa49
Jul 14 15:21:18.401846 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:18.401822    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15614b5c_a711_470b_92b6_7ad03154f6be.slice/crio-5d65c02d9e7f2fbcc14324ab3b5777bfe52a2435e98d17339ec5782b488d2112 WatchSource:0}: Error finding container 5d65c02d9e7f2fbcc14324ab3b5777bfe52a2435e98d17339ec5782b488d2112: Status 404 returned error can't find the container with id 5d65c02d9e7f2fbcc14324ab3b5777bfe52a2435e98d17339ec5782b488d2112
Jul 14 15:21:18.403420 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:18.403395    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46dbf1e8_eadf_4cf0_ac0f_5bfb770bcff5.slice/crio-6cd863ed57c3309f121c1d3d6c54373f2b2f223d3e5c15a5c5a2801e818b4f80 WatchSource:0}: Error finding container 6cd863ed57c3309f121c1d3d6c54373f2b2f223d3e5c15a5c5a2801e818b4f80: Status 404 returned error can't find the container with id 6cd863ed57c3309f121c1d3d6c54373f2b2f223d3e5c15a5c5a2801e818b4f80
Jul 14 15:21:18.404781 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:18.404739    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0e4d6e_5859_4df9_9f22_d5e18660a104.slice/crio-73e1859d378a94d69aaa571dcc2e59704963f2607c7f0380b39d3a8cb15b7a73 WatchSource:0}: Error finding container 73e1859d378a94d69aaa571dcc2e59704963f2607c7f0380b39d3a8cb15b7a73: Status 404 returned error can't find the container with id 73e1859d378a94d69aaa571dcc2e59704963f2607c7f0380b39d3a8cb15b7a73
Jul 14 15:21:18.406434 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:18.406412    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2af3c99_9843_496a_823b_f8a97a1dcd9d.slice/crio-ab6b79fba7e36ab5364a428e3c74848d1db99747a5efa533e8462691340463d3 WatchSource:0}: Error finding container ab6b79fba7e36ab5364a428e3c74848d1db99747a5efa533e8462691340463d3: Status 404 returned error can't find the container with id ab6b79fba7e36ab5364a428e3c74848d1db99747a5efa533e8462691340463d3
Jul 14 15:21:18.727785 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.727539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqm7w\" (UniqueName: \"kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w\") pod \"network-check-target-w82g9\" (UID: \"bf1192e2-ad6e-4efc-964e-3c31e509c8e6\") " pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:18.727953 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.727809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:18.727953 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.727703    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 15:21:18.727953 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.727899    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 15:21:18.727953 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.727914    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bqm7w for pod openshift-network-diagnostics/network-check-target-w82g9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 15:21:18.728188 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.727961    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 15:21:18.728188 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.727980    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w podName:bf1192e2-ad6e-4efc-964e-3c31e509c8e6 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:19.727959368 +0000 UTC m=+4.069225273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bqm7w" (UniqueName: "kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w") pod "network-check-target-w82g9" (UID: "bf1192e2-ad6e-4efc-964e-3c31e509c8e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 15:21:18.728188 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.728012    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret podName:d3ae120a-8040-483a-8da6-090ef1e80265 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:19.727996479 +0000 UTC m=+4.069262386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret") pod "global-pull-secret-syncer-w8dm9" (UID: "d3ae120a-8040-483a-8da6-090ef1e80265") : object "kube-system"/"original-pull-secret" not registered
Jul 14 15:21:18.828948 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:18.828826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs\") pod \"network-metrics-daemon-frtx9\" (UID: \"b42135f6-4895-4ca1-8086-1ae81b8006c8\") " pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:18.829110 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.828955    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 15:21:18.829110 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:18.829036    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs podName:b42135f6-4895-4ca1-8086-1ae81b8006c8 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:19.829013698 +0000 UTC m=+4.170279600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs") pod "network-metrics-daemon-frtx9" (UID: "b42135f6-4895-4ca1-8086-1ae81b8006c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 15:21:19.055402 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.055284    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 15:16:18 +0000 UTC" deadline="2028-03-12 03:22:17.639529985 +0000 UTC"
Jul 14 15:21:19.055402 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.055340    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14556h0m58.584195874s"
Jul 14 15:21:19.221776 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.220651    2563 generic.go:356] "Generic (PLEG): container finished" podID="a95379fcdfcd10b1894525c55897d3ca" containerID="1609c78944487ef6c7182e8d156c02a577e9c5245df36524c53f221d52e10cf4" exitCode=0
Jul 14 15:21:19.221776 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.220750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal" event={"ID":"a95379fcdfcd10b1894525c55897d3ca","Type":"ContainerDied","Data":"1609c78944487ef6c7182e8d156c02a577e9c5245df36524c53f221d52e10cf4"}
Jul 14 15:21:19.228059 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.228028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln" event={"ID":"544bf2b6-164b-4c05-8255-016265e27df5","Type":"ContainerStarted","Data":"b36654d1675b2cdc9e10b9266882c6b74bc10f0761671f7f1a63ba581884ef35"}
Jul 14 15:21:19.232835 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.232714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2" event={"ID":"7e71167e-01b8-42b0-9403-8ae910ec77bf","Type":"ContainerStarted","Data":"897e9b95109f3497d8a7d5c2b3484d782257f75595461e1276a54cf458da725e"}
Jul 14 15:21:19.236442 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.236358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7phx7" event={"ID":"d2af3c99-9843-496a-823b-f8a97a1dcd9d","Type":"ContainerStarted","Data":"ab6b79fba7e36ab5364a428e3c74848d1db99747a5efa533e8462691340463d3"}
Jul 14 15:21:19.247729 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.247698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rtlt8" event={"ID":"15614b5c-a711-470b-92b6-7ad03154f6be","Type":"ContainerStarted","Data":"5d65c02d9e7f2fbcc14324ab3b5777bfe52a2435e98d17339ec5782b488d2112"}
Jul 14 15:21:19.250141 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.250055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpqz" event={"ID":"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7","Type":"ContainerStarted","Data":"f7b57c18abbf12a8c93b161ad7fa52ea75438cbbb5c6f1ca778085e55e8aaa49"}
Jul 14 15:21:19.254593 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.254493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l2dj2" event={"ID":"93ac7979-c05b-4b5c-8277-d9098a69b8b5","Type":"ContainerStarted","Data":"7b6703ec5e61f4f3ea2770eee506b850081a7bdde93ac2fe2d612c36b678021e"}
Jul 14 15:21:19.258808 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.258779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9f2sc" event={"ID":"b9623835-e777-4e7d-95be-70ce05205902","Type":"ContainerStarted","Data":"7e269dc979bda1bafe680be4f99105a03a85f57a862cd9dec2341ced7a2368c4"}
Jul 14 15:21:19.265950 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.265901    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bwj9q" event={"ID":"46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5","Type":"ContainerStarted","Data":"6cd863ed57c3309f121c1d3d6c54373f2b2f223d3e5c15a5c5a2801e818b4f80"}
Jul 14 15:21:19.269896 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.269870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tjprj" event={"ID":"0e0e4d6e-5859-4df9-9f22-d5e18660a104","Type":"ContainerStarted","Data":"73e1859d378a94d69aaa571dcc2e59704963f2607c7f0380b39d3a8cb15b7a73"}
Jul 14 15:21:19.456927 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:21:19.456890    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d95b182bc9232c53452fc18d92789eb.slice/crio-effdfd6613707d722639301a76a6bcb05136a10b148f1a704b10c4e37e4417d8 WatchSource:0}: Error finding container effdfd6613707d722639301a76a6bcb05136a10b148f1a704b10c4e37e4417d8: Status 404 returned error can't find the container with id effdfd6613707d722639301a76a6bcb05136a10b148f1a704b10c4e37e4417d8
Jul 14 15:21:19.736087 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.736033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqm7w\" (UniqueName: \"kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w\") pod \"network-check-target-w82g9\" (UID: \"bf1192e2-ad6e-4efc-964e-3c31e509c8e6\") " pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:19.736087 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.736084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:19.736349 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:19.736232    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 15:21:19.736349 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:19.736318    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret podName:d3ae120a-8040-483a-8da6-090ef1e80265 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:21.736296854 +0000 UTC m=+6.077562758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret") pod "global-pull-secret-syncer-w8dm9" (UID: "d3ae120a-8040-483a-8da6-090ef1e80265") : object "kube-system"/"original-pull-secret" not registered
Jul 14 15:21:19.736737 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:19.736714    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 15:21:19.736846 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:19.736746    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 15:21:19.736846 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:19.736778    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bqm7w for pod openshift-network-diagnostics/network-check-target-w82g9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 15:21:19.736846 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:19.736834    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w podName:bf1192e2-ad6e-4efc-964e-3c31e509c8e6 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:21.736816969 +0000 UTC m=+6.078082857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bqm7w" (UniqueName: "kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w") pod "network-check-target-w82g9" (UID: "bf1192e2-ad6e-4efc-964e-3c31e509c8e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 15:21:19.837059 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:19.837014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs\") pod \"network-metrics-daemon-frtx9\" (UID: \"b42135f6-4895-4ca1-8086-1ae81b8006c8\") " pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:19.837252 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:19.837232    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 15:21:19.837312 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:19.837305    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs podName:b42135f6-4895-4ca1-8086-1ae81b8006c8 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:21.837284081 +0000 UTC m=+6.178549977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs") pod "network-metrics-daemon-frtx9" (UID: "b42135f6-4895-4ca1-8086-1ae81b8006c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 15:21:20.057612 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:20.055688    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 15:16:18 +0000 UTC" deadline="2028-01-22 11:08:09.391421688 +0000 UTC"
Jul 14 15:21:20.057612 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:20.055733    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13363h46m49.335693973s"
Jul 14 15:21:20.064278 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:20.063995    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 15:21:20.213632 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:20.213176    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:20.213632 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:20.213334    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:20.213942 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:20.213781    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:20.213942 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:20.213905    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:20.214049 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:20.213971    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:20.214049 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:20.214037    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:20.290998 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:20.290953    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-210.ec2.internal" event={"ID":"3d95b182bc9232c53452fc18d92789eb","Type":"ContainerStarted","Data":"effdfd6613707d722639301a76a6bcb05136a10b148f1a704b10c4e37e4417d8"}
Jul 14 15:21:20.305657 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:20.305610    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal" event={"ID":"a95379fcdfcd10b1894525c55897d3ca","Type":"ContainerStarted","Data":"0126262edb9887b9418eab2bc600c4921f0dba79cbda0765054547f31ce26fe1"}
Jul 14 15:21:21.759241 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:21.759196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqm7w\" (UniqueName: \"kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w\") pod \"network-check-target-w82g9\" (UID: \"bf1192e2-ad6e-4efc-964e-3c31e509c8e6\") " pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:21.759746 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:21.759255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:21.759746 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:21.759399    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 15:21:21.759746 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:21.759419    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 15:21:21.759746 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:21.759439    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 15:21:21.759746 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:21.759453    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bqm7w for pod openshift-network-diagnostics/network-check-target-w82g9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 15:21:21.759746 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:21.759464    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret podName:d3ae120a-8040-483a-8da6-090ef1e80265 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:25.759445811 +0000 UTC m=+10.100711713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret") pod "global-pull-secret-syncer-w8dm9" (UID: "d3ae120a-8040-483a-8da6-090ef1e80265") : object "kube-system"/"original-pull-secret" not registered
Jul 14 15:21:21.759746 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:21.759489    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w podName:bf1192e2-ad6e-4efc-964e-3c31e509c8e6 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:25.75947817 +0000 UTC m=+10.100744070 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bqm7w" (UniqueName: "kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w") pod "network-check-target-w82g9" (UID: "bf1192e2-ad6e-4efc-964e-3c31e509c8e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 15:21:21.860012 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:21.859973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs\") pod \"network-metrics-daemon-frtx9\" (UID: \"b42135f6-4895-4ca1-8086-1ae81b8006c8\") " pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:21.860215 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:21.860125    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 15:21:21.860215 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:21.860207    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs podName:b42135f6-4895-4ca1-8086-1ae81b8006c8 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:25.860185199 +0000 UTC m=+10.201451101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs") pod "network-metrics-daemon-frtx9" (UID: "b42135f6-4895-4ca1-8086-1ae81b8006c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 15:21:22.213569 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:22.213460    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:22.213569 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:22.213522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:22.213792 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:22.213651    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:22.213792 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:22.213731    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:22.213861 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:22.213815    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:22.213955 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:22.213931    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:24.212136 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:24.211240    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:24.212136 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:24.211385    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:24.212136 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:24.211855    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:24.212136 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:24.211954    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:24.212136 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:24.212005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:24.212136 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:24.212086    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:25.795056 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:25.795012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqm7w\" (UniqueName: \"kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w\") pod \"network-check-target-w82g9\" (UID: \"bf1192e2-ad6e-4efc-964e-3c31e509c8e6\") " pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:25.795512 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:25.795069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:25.795512 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:25.795229    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 15:21:25.795512 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:25.795295    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret podName:d3ae120a-8040-483a-8da6-090ef1e80265 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:33.795282243 +0000 UTC m=+18.136548129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret") pod "global-pull-secret-syncer-w8dm9" (UID: "d3ae120a-8040-483a-8da6-090ef1e80265") : object "kube-system"/"original-pull-secret" not registered
Jul 14 15:21:25.795739 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:25.795667    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 15:21:25.795739 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:25.795685    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 15:21:25.795739 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:25.795699    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bqm7w for pod openshift-network-diagnostics/network-check-target-w82g9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 15:21:25.795739 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:25.795737    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w podName:bf1192e2-ad6e-4efc-964e-3c31e509c8e6 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:33.795724356 +0000 UTC m=+18.136990256 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bqm7w" (UniqueName: "kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w") pod "network-check-target-w82g9" (UID: "bf1192e2-ad6e-4efc-964e-3c31e509c8e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 15:21:25.896808 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:25.896224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs\") pod \"network-metrics-daemon-frtx9\" (UID: \"b42135f6-4895-4ca1-8086-1ae81b8006c8\") " pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:25.896808 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:25.896372    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 15:21:25.896808 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:25.896433    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs podName:b42135f6-4895-4ca1-8086-1ae81b8006c8 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:33.896418602 +0000 UTC m=+18.237684488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs") pod "network-metrics-daemon-frtx9" (UID: "b42135f6-4895-4ca1-8086-1ae81b8006c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 15:21:26.211755 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:26.211241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:26.211755 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:26.211343    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:26.211755 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:26.211353    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:26.211755 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:26.211241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:26.211755 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:26.211455    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:26.211755 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:26.211594    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:28.210577 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:28.210539    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:28.211024 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:28.210682    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:28.211086 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:28.211062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:28.211176 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:28.211156    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:28.211284 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:28.211251    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:28.211374 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:28.211355    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:30.210242 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:30.209850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:30.210242 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:30.209850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:30.210242 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:30.209959    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:30.210242 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:30.209970    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:30.210242 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:30.210050    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:30.210242 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:30.210128    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:32.210823 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:32.210786    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:32.211307 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:32.210784    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:32.211307 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:32.210931    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:32.211307 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:32.210802    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:32.211307 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:32.211012    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:32.211307 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:32.211105    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:33.859539 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:33.859490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqm7w\" (UniqueName: \"kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w\") pod \"network-check-target-w82g9\" (UID: \"bf1192e2-ad6e-4efc-964e-3c31e509c8e6\") " pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:33.860003 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:33.859559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:33.860003 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:33.859656    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 15:21:33.860003 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:33.859685    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 15:21:33.860003 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:33.859699    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bqm7w for pod openshift-network-diagnostics/network-check-target-w82g9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 15:21:33.860003 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:33.859721    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 15:21:33.860003 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:33.859760    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w podName:bf1192e2-ad6e-4efc-964e-3c31e509c8e6 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:49.859740589 +0000 UTC m=+34.201006486 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bqm7w" (UniqueName: "kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w") pod "network-check-target-w82g9" (UID: "bf1192e2-ad6e-4efc-964e-3c31e509c8e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 15:21:33.860003 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:33.859784    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret podName:d3ae120a-8040-483a-8da6-090ef1e80265 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:49.859770096 +0000 UTC m=+34.201036003 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret") pod "global-pull-secret-syncer-w8dm9" (UID: "d3ae120a-8040-483a-8da6-090ef1e80265") : object "kube-system"/"original-pull-secret" not registered
Jul 14 15:21:33.960704 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:33.960665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs\") pod \"network-metrics-daemon-frtx9\" (UID: \"b42135f6-4895-4ca1-8086-1ae81b8006c8\") " pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:33.960860 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:33.960780    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 15:21:33.960860 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:33.960854    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs podName:b42135f6-4895-4ca1-8086-1ae81b8006c8 nodeName:}" failed. No retries permitted until 2026-07-14 15:21:49.960833399 +0000 UTC m=+34.302099294 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs") pod "network-metrics-daemon-frtx9" (UID: "b42135f6-4895-4ca1-8086-1ae81b8006c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 15:21:34.210414 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:34.210382    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:34.210414 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:34.210414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:34.210597 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:34.210383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:34.210597 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:34.210496    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:34.210676 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:34.210610    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:34.210795 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:34.210771    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:36.212304 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:36.211810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:36.212304 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:36.211926    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:36.212304 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:36.211925    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:36.212304 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:36.212057    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:36.212304 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:36.212077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:36.212304 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:36.212198    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:38.210112 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:38.210084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:38.210112 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:38.210101    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:38.210112 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:38.210111    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:38.210859 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:38.210208    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:38.210859 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:38.210287    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:38.210859 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:38.210369    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:39.341627 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.341363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-210.ec2.internal" event={"ID":"3d95b182bc9232c53452fc18d92789eb","Type":"ContainerStarted","Data":"b5e12d56c7aeefb31733bacceca23c4a0a5b327e23ca7d9f861748122175535f"}
Jul 14 15:21:39.342782 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.342761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln" event={"ID":"544bf2b6-164b-4c05-8255-016265e27df5","Type":"ContainerStarted","Data":"924ebabca9d8aced660e265c2492f28850559235ce91e413936a7631d8a96021"}
Jul 14 15:21:39.345097 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.345079    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:21:39.345374 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.345356    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e71167e-01b8-42b0-9403-8ae910ec77bf" containerID="b8f520b69b9aa717f1c88c62a707dedb88f5c3a26fc1a4edc572a01289d6ab8f" exitCode=1
Jul 14 15:21:39.345436 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.345418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2" event={"ID":"7e71167e-01b8-42b0-9403-8ae910ec77bf","Type":"ContainerStarted","Data":"dc905ea522bb6b623fe868fee76b1517f887f25460e0ae428590e20fb376d1fd"}
Jul 14 15:21:39.345475 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.345467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2" event={"ID":"7e71167e-01b8-42b0-9403-8ae910ec77bf","Type":"ContainerStarted","Data":"4c8de999f2f8d0e1ecdf52302de5075bd6521bf84e1bab61b71da48b51473377"}
Jul 14 15:21:39.345538 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.345497    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2" event={"ID":"7e71167e-01b8-42b0-9403-8ae910ec77bf","Type":"ContainerStarted","Data":"b9106379e4b24307d5f6dbbd26f92fa4e51a29a133275fcfd6aa170d37f61494"}
Jul 14 15:21:39.345577 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.345538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2" event={"ID":"7e71167e-01b8-42b0-9403-8ae910ec77bf","Type":"ContainerStarted","Data":"e4f97604e0478eee13fa5204b03cb10c8643f8f0285b70938c91958f76c20f1d"}
Jul 14 15:21:39.345577 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.345565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2" event={"ID":"7e71167e-01b8-42b0-9403-8ae910ec77bf","Type":"ContainerDied","Data":"b8f520b69b9aa717f1c88c62a707dedb88f5c3a26fc1a4edc572a01289d6ab8f"}
Jul 14 15:21:39.345661 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.345592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2" event={"ID":"7e71167e-01b8-42b0-9403-8ae910ec77bf","Type":"ContainerStarted","Data":"fad665dea4b14b31dbd518832aacc34eef7934f71baf7658934c97e365c910a8"}
Jul 14 15:21:39.346607 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.346587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7phx7" event={"ID":"d2af3c99-9843-496a-823b-f8a97a1dcd9d","Type":"ContainerStarted","Data":"9540a38ad1375bd239d54e49a5cf8c7690580afb54c8da3850ddd21c3857cb94"}
Jul 14 15:21:39.347929 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.347911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rtlt8" event={"ID":"15614b5c-a711-470b-92b6-7ad03154f6be","Type":"ContainerStarted","Data":"7639c05e1700a6a40700534118f47da7ebc3965edb0783507feab22977a5c381"}
Jul 14 15:21:39.349203 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.349185    2563 generic.go:356] "Generic (PLEG): container finished" podID="5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7" containerID="d3eb1d5a94203af1944e1026d0ceec61fc7dc24ba1f0f19a9f55d8f6dec99390" exitCode=0
Jul 14 15:21:39.349285 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.349241    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpqz" event={"ID":"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7","Type":"ContainerDied","Data":"d3eb1d5a94203af1944e1026d0ceec61fc7dc24ba1f0f19a9f55d8f6dec99390"}
Jul 14 15:21:39.350588 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.350568    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l2dj2" event={"ID":"93ac7979-c05b-4b5c-8277-d9098a69b8b5","Type":"ContainerStarted","Data":"6a50c9219e702fe82dedfa0d28ee75659b52f8b67cb69bcb19d8ed6afec3d65c"}
Jul 14 15:21:39.351894 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.351874    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9f2sc" event={"ID":"b9623835-e777-4e7d-95be-70ce05205902","Type":"ContainerStarted","Data":"27d61b557643018e94ffea5e47498cf56000b79b7fe87de03607c90ec9eaa523"}
Jul 14 15:21:39.353187 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.353170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tjprj" event={"ID":"0e0e4d6e-5859-4df9-9f22-d5e18660a104","Type":"ContainerStarted","Data":"c1a1085b764924a9a6638bd6c21df24ae350824c657fed41315eacd6a5e1128d"}
Jul 14 15:21:39.353964 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.353925    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-210.ec2.internal" podStartSLOduration=22.353897801 podStartE2EDuration="22.353897801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:20.325319517 +0000 UTC m=+4.666585432" watchObservedRunningTime="2026-07-14 15:21:39.353897801 +0000 UTC m=+23.695163702"
Jul 14 15:21:39.367245 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.367205    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rtlt8" podStartSLOduration=11.326461668 podStartE2EDuration="23.367195428s" totalImagesPullingTime="12.04073376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:39.36717208 +0000 UTC m=+23.708437986" watchObservedRunningTime="2026-07-14 15:21:39.367195428 +0000 UTC m=+23.708461333"
Jul 14 15:21:39.367338 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.367301    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-210.ec2.internal" podStartSLOduration=22.367294529 podStartE2EDuration="22.367294529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:39.354212528 +0000 UTC m=+23.695478433" watchObservedRunningTime="2026-07-14 15:21:39.367294529 +0000 UTC m=+23.708560435"
Jul 14 15:21:39.379932 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.379898    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7phx7" podStartSLOduration=3.478775606 podStartE2EDuration="23.37988749s" totalImagesPullingTime="19.901111884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:39.379659881 +0000 UTC m=+23.720925811" watchObservedRunningTime="2026-07-14 15:21:39.37988749 +0000 UTC m=+23.721153425"
Jul 14 15:21:39.415025 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.414984    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tjprj" podStartSLOduration=2.489911216 podStartE2EDuration="22.41496976s" totalImagesPullingTime="19.925058544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:39.414736096 +0000 UTC m=+23.756002003" watchObservedRunningTime="2026-07-14 15:21:39.41496976 +0000 UTC m=+23.756235666"
Jul 14 15:21:39.432695 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.431418    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l2dj2" podStartSLOduration=3.444472754 podStartE2EDuration="23.431399581s" totalImagesPullingTime="19.986926827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:39.429862814 +0000 UTC m=+23.771128721" watchObservedRunningTime="2026-07-14 15:21:39.431399581 +0000 UTC m=+23.772665487"
Jul 14 15:21:39.444290 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.444250    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9f2sc" podStartSLOduration=2.502657825 podStartE2EDuration="22.444237241s" totalImagesPullingTime="19.941579416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:39.444039429 +0000 UTC m=+23.785305351" watchObservedRunningTime="2026-07-14 15:21:39.444237241 +0000 UTC m=+23.785503148"
Jul 14 15:21:39.748103 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:39.748078    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 14 15:21:40.074948 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:40.074793    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T15:21:39.748097269Z","UUID":"f58004aa-5d59-42d3-a513-06b7771d32b6","Handler":null,"Name":"","Endpoint":""}
Jul 14 15:21:40.076327 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:40.076304    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 14 15:21:40.076463 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:40.076349    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 14 15:21:40.209947 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:40.209915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:40.210176 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:40.209915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:40.210176 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:40.210063    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:40.210176 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:40.209915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:40.210176 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:40.210152    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:40.210440 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:40.210198    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:40.358048 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:40.357963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln" event={"ID":"544bf2b6-164b-4c05-8255-016265e27df5","Type":"ContainerStarted","Data":"5fcf16e33f7201498fc9805ea397834c49f5ff6da0c9ab751fe3c2e9d9abb4f2"}
Jul 14 15:21:40.359736 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:40.359708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bwj9q" event={"ID":"46dbf1e8-eadf-4cf0-ac0f-5bfb770bcff5","Type":"ContainerStarted","Data":"3b6e91f96de243f6237b728d320cf89eac574000f3b31d5dbc6eadfe2ddf14d7"}
Jul 14 15:21:40.374898 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:40.374847    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bwj9q" podStartSLOduration=3.470959894 podStartE2EDuration="23.374831082s" totalImagesPullingTime="19.903871188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:40.37470508 +0000 UTC m=+24.715970989" watchObservedRunningTime="2026-07-14 15:21:40.374831082 +0000 UTC m=+24.716096987"
Jul 14 15:21:41.364975 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:41.364671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln" event={"ID":"544bf2b6-164b-4c05-8255-016265e27df5","Type":"ContainerStarted","Data":"3d21b36e5cf80b28b8ffd816e3a77882c5bec53d5bfe59452c8de1f29691c509"}
Jul 14 15:21:41.369097 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:41.369071    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:21:41.369552 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:41.369525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2" event={"ID":"7e71167e-01b8-42b0-9403-8ae910ec77bf","Type":"ContainerStarted","Data":"12e6e4026443ea1ef0025790368ca8767d6107b92fd7cb9082cc4d2a80075936"}
Jul 14 15:21:41.383270 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:41.383227    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m96ln" podStartSLOduration=2.120767448 podStartE2EDuration="24.383203938s" totalImagesPullingTime="22.26243649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:41.38313118 +0000 UTC m=+25.724397111" watchObservedRunningTime="2026-07-14 15:21:41.383203938 +0000 UTC m=+25.724469844"
Jul 14 15:21:42.035230 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:42.035152    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rtlt8"
Jul 14 15:21:42.035891 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:42.035867    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rtlt8"
Jul 14 15:21:42.210873 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:42.210835    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:42.210873 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:42.210871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:42.211105 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:42.210835    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:42.211105 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:42.210981    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:42.211269 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:42.211238    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:42.211390 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:42.211368    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:42.373038 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:42.372813    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rtlt8"
Jul 14 15:21:42.373497 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:42.373362    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rtlt8"
Jul 14 15:21:43.377358 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:43.377336    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:21:43.378144 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:43.377783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2" event={"ID":"7e71167e-01b8-42b0-9403-8ae910ec77bf","Type":"ContainerStarted","Data":"e4f706fd9ca66c8a9aef19612b37394ceebe7f2d0ab17c46cbc9382c71d24782"}
Jul 14 15:21:44.210266 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:44.210036    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:44.210419 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:44.210064    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:44.210419 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:44.210311    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:44.210419 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:44.210382    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:44.210419 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:44.210111    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:44.210555 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:44.210463    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:44.382011 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:44.381976    2563 generic.go:356] "Generic (PLEG): container finished" podID="5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7" containerID="93960ee5460e2f25ff3fe9aa4bc8054469ff53fcfd3259658a2497a7314c136e" exitCode=0
Jul 14 15:21:44.382442 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:44.382063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpqz" event={"ID":"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7","Type":"ContainerDied","Data":"93960ee5460e2f25ff3fe9aa4bc8054469ff53fcfd3259658a2497a7314c136e"}
Jul 14 15:21:44.383341 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:44.382692    2563 scope.go:117] "RemoveContainer" containerID="b8f520b69b9aa717f1c88c62a707dedb88f5c3a26fc1a4edc572a01289d6ab8f"
Jul 14 15:21:45.388585 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.388557    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:21:45.389072 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.389043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2" event={"ID":"7e71167e-01b8-42b0-9403-8ae910ec77bf","Type":"ContainerStarted","Data":"2766f4516a36c135b7ec998731099e5f795b6879397a01f818ab2c1368990261"}
Jul 14 15:21:45.389585 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.389542    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:45.389669 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.389623    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:45.389669 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.389669    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:45.391752 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.391716    2563 generic.go:356] "Generic (PLEG): container finished" podID="5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7" containerID="7e5b7de10a80067de54c88034e1a87e9f8bdab449d7e63abf79f741eb2f25527" exitCode=0
Jul 14 15:21:45.391861 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.391764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpqz" event={"ID":"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7","Type":"ContainerDied","Data":"7e5b7de10a80067de54c88034e1a87e9f8bdab449d7e63abf79f741eb2f25527"}
Jul 14 15:21:45.407574 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.407477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:45.407774 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.407760    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:21:45.421924 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.421883    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2" podStartSLOduration=8.394028116 podStartE2EDuration="28.421867915s" totalImagesPullingTime="20.027839799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:45.421367137 +0000 UTC m=+29.762633060" watchObservedRunningTime="2026-07-14 15:21:45.421867915 +0000 UTC m=+29.763133820"
Jul 14 15:21:45.599041 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.598969    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w82g9"]
Jul 14 15:21:45.599171 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.599073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:45.599238 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:45.599219    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:45.602316 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.602286    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-frtx9"]
Jul 14 15:21:45.602485 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.602465    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:45.602628 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:45.602602    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:45.602944 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.602886    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w8dm9"]
Jul 14 15:21:45.603045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:45.602969    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:45.603107 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:45.603041    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:47.210696 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:47.210518    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:47.211181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:47.210518    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:47.211181 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:47.210769    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:47.211181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:47.210518    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:47.211181 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:47.210835    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:47.211181 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:47.210921    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:47.399355 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:47.399322    2563 generic.go:356] "Generic (PLEG): container finished" podID="5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7" containerID="37976aec0c9c2c046907cc7ca71e2da423be30218769e0374a06555f5d60a4ea" exitCode=0
Jul 14 15:21:47.399537 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:47.399403    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpqz" event={"ID":"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7","Type":"ContainerDied","Data":"37976aec0c9c2c046907cc7ca71e2da423be30218769e0374a06555f5d60a4ea"}
Jul 14 15:21:47.892701 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:47.892674    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9f2sc_b9623835-e777-4e7d-95be-70ce05205902/dns-node-resolver/0.log"
Jul 14 15:21:48.862462 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:48.862434    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7phx7_d2af3c99-9843-496a-823b-f8a97a1dcd9d/node-ca/0.log"
Jul 14 15:21:49.210060 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:49.209970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:49.210060 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:49.209978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:49.210318 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:49.210095    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:49.210318 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:49.210143    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:49.210318 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:49.210208    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:49.210318 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:49.210263    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:49.878705 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:49.878668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqm7w\" (UniqueName: \"kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w\") pod \"network-check-target-w82g9\" (UID: \"bf1192e2-ad6e-4efc-964e-3c31e509c8e6\") " pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:49.878705 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:49.878713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:49.879276 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:49.878830    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 15:21:49.879276 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:49.878873    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 15:21:49.879276 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:49.878895    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret podName:d3ae120a-8040-483a-8da6-090ef1e80265 nodeName:}" failed. No retries permitted until 2026-07-14 15:22:21.878875851 +0000 UTC m=+66.220141743 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret") pod "global-pull-secret-syncer-w8dm9" (UID: "d3ae120a-8040-483a-8da6-090ef1e80265") : object "kube-system"/"original-pull-secret" not registered
Jul 14 15:21:49.879276 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:49.878902    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 15:21:49.879276 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:49.878916    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bqm7w for pod openshift-network-diagnostics/network-check-target-w82g9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 15:21:49.879276 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:49.878975    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w podName:bf1192e2-ad6e-4efc-964e-3c31e509c8e6 nodeName:}" failed. No retries permitted until 2026-07-14 15:22:21.878959884 +0000 UTC m=+66.220225776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bqm7w" (UniqueName: "kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w") pod "network-check-target-w82g9" (UID: "bf1192e2-ad6e-4efc-964e-3c31e509c8e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 15:21:49.980031 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:49.979982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs\") pod \"network-metrics-daemon-frtx9\" (UID: \"b42135f6-4895-4ca1-8086-1ae81b8006c8\") " pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:49.980209 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:49.980141    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 15:21:49.980299 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:49.980209    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs podName:b42135f6-4895-4ca1-8086-1ae81b8006c8 nodeName:}" failed. No retries permitted until 2026-07-14 15:22:21.980190823 +0000 UTC m=+66.321456711 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs") pod "network-metrics-daemon-frtx9" (UID: "b42135f6-4895-4ca1-8086-1ae81b8006c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 15:21:51.210251 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:51.210213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:51.210251 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:51.210238    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:51.210810 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:51.210328    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:51.210810 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:51.210441    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:51.210810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:51.210495    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:51.210810 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:51.210583    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:53.209856 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:53.209823    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:53.210279 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:53.209944    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:53.210279 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:53.209960    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:53.210279 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:53.209984    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:53.210279 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:53.210083    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:53.210279 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:53.210161    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:53.414671 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:53.414637    2563 generic.go:356] "Generic (PLEG): container finished" podID="5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7" containerID="4687679cb220994aee3318f307a5b3c2dd5e4dfb32d4f9cf903ce527e0c1a16c" exitCode=0
Jul 14 15:21:53.414817 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:53.414691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpqz" event={"ID":"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7","Type":"ContainerDied","Data":"4687679cb220994aee3318f307a5b3c2dd5e4dfb32d4f9cf903ce527e0c1a16c"}
Jul 14 15:21:54.419336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:54.419300    2563 generic.go:356] "Generic (PLEG): container finished" podID="5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7" containerID="613e975329d79b47897ac5d9c6722b7a60ef999cc5b505d003dc836bc2ee5222" exitCode=0
Jul 14 15:21:54.419816 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:54.419360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpqz" event={"ID":"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7","Type":"ContainerDied","Data":"613e975329d79b47897ac5d9c6722b7a60ef999cc5b505d003dc836bc2ee5222"}
Jul 14 15:21:55.210414 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:55.210238    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:55.210595 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:55.210254    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:55.210595 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:55.210492    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:55.210595 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:55.210288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:55.210713 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:55.210616    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:55.210713 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:55.210694    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:55.427213 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:55.427170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpqz" event={"ID":"5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7","Type":"ContainerStarted","Data":"76149dec5fee4f74b042ae76cccf2dd141a64754bc4606f9ce37944c3845e799"}
Jul 14 15:21:55.452050 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:55.452004    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bxpqz" podStartSLOduration=7.592778114 podStartE2EDuration="39.451990776s" totalImagesPullingTime="31.859212662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:21:55.449961261 +0000 UTC m=+39.791227186" watchObservedRunningTime="2026-07-14 15:21:55.451990776 +0000 UTC m=+39.793256683"
Jul 14 15:21:57.209799 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:57.209764    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:57.210256 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:57.209769    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:57.210256 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:57.209869    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:21:57.210256 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:57.209893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:57.210256 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:57.210013    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:57.210256 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:57.210063    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:59.210231 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:59.210197    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:21:59.210626 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:59.210293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:21:59.210626 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:21:59.210304    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:21:59.210626 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:59.210398    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:21:59.210626 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:59.210474    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:21:59.210626 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:21:59.210301    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:22:01.210544 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:01.210486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:22:01.210544 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:01.210530    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:22:01.211012 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:01.210551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:22:01.211012 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:01.210633    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:22:01.211012 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:01.210675    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:22:01.211012 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:01.210755    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:22:03.210476 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:03.210433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:22:03.210476 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:03.210462    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:22:03.210930 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:03.210434    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:22:03.210930 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:03.210572    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:22:03.210930 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:03.210626    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:22:03.210930 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:03.210684    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:22:05.210709 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:05.210671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:22:05.210709 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:05.210700    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:22:05.211154 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:05.210688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:22:05.211154 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:05.210792    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frtx9" podUID="b42135f6-4895-4ca1-8086-1ae81b8006c8"
Jul 14 15:22:05.211154 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:05.210844    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w82g9" podUID="bf1192e2-ad6e-4efc-964e-3c31e509c8e6"
Jul 14 15:22:05.211154 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:05.210900    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8dm9" podUID="d3ae120a-8040-483a-8da6-090ef1e80265"
Jul 14 15:22:06.955039 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:06.955006    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-210.ec2.internal" event="NodeReady"
Jul 14 15:22:06.955433 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:06.955150    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 14 15:22:07.013617 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.013572    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-glxgf"]
Jul 14 15:22:07.034930 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.034860    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r444t"]
Jul 14 15:22:07.035143 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.035103    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-glxgf"
Jul 14 15:22:07.038079 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.038061    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p6wcc\""
Jul 14 15:22:07.038354 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.038341    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 14 15:22:07.038582 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.038566    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 14 15:22:07.038622 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.038583    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 14 15:22:07.058722 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.058700    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-glxgf"]
Jul 14 15:22:07.058722 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.058734    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r444t"]
Jul 14 15:22:07.058904 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.058759    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vmlvz"]
Jul 14 15:22:07.058904 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.058820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:07.061930 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.061911    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xtkj9\""
Jul 14 15:22:07.062016 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.061909    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 14 15:22:07.062016 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.061965    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 14 15:22:07.082730 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.082667    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vmlvz"]
Jul 14 15:22:07.082821 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.082778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.085815 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.085798    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 14 15:22:07.085913 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.085821    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 14 15:22:07.085913 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.085799    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bllnj\""
Jul 14 15:22:07.085913 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.085895    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 14 15:22:07.086072 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.085922    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 14 15:22:07.097227 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.097209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdwxn\" (UniqueName: \"kubernetes.io/projected/a85dd215-4b19-4732-baf4-d02c27595e38-kube-api-access-hdwxn\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.097326 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.097245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a85dd215-4b19-4732-baf4-d02c27595e38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.097326 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.097265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f14f969b-df8a-4443-951e-4c9318e2362c-cert\") pod \"ingress-canary-glxgf\" (UID: \"f14f969b-df8a-4443-951e-4c9318e2362c\") " pod="openshift-ingress-canary/ingress-canary-glxgf"
Jul 14 15:22:07.097326 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.097284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtrm\" (UniqueName: \"kubernetes.io/projected/442ff21f-3879-494d-a460-be0c043a5aa8-kube-api-access-svtrm\") pod \"dns-default-r444t\" (UID: \"442ff21f-3879-494d-a460-be0c043a5aa8\") " pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:07.097423 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.097352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/442ff21f-3879-494d-a460-be0c043a5aa8-metrics-tls\") pod \"dns-default-r444t\" (UID: \"442ff21f-3879-494d-a460-be0c043a5aa8\") " pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:07.097423 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.097388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a85dd215-4b19-4732-baf4-d02c27595e38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.097423 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.097411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a85dd215-4b19-4732-baf4-d02c27595e38-data-volume\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.097528 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.097428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/442ff21f-3879-494d-a460-be0c043a5aa8-config-volume\") pod \"dns-default-r444t\" (UID: \"442ff21f-3879-494d-a460-be0c043a5aa8\") " pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:07.097528 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.097460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442ff21f-3879-494d-a460-be0c043a5aa8-tmp-dir\") pod \"dns-default-r444t\" (UID: \"442ff21f-3879-494d-a460-be0c043a5aa8\") " pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:07.097528 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.097487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wct5q\" (UniqueName: \"kubernetes.io/projected/f14f969b-df8a-4443-951e-4c9318e2362c-kube-api-access-wct5q\") pod \"ingress-canary-glxgf\" (UID: \"f14f969b-df8a-4443-951e-4c9318e2362c\") " pod="openshift-ingress-canary/ingress-canary-glxgf"
Jul 14 15:22:07.097620 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.097545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a85dd215-4b19-4732-baf4-d02c27595e38-crio-socket\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.198116 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdwxn\" (UniqueName: \"kubernetes.io/projected/a85dd215-4b19-4732-baf4-d02c27595e38-kube-api-access-hdwxn\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.198287 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a85dd215-4b19-4732-baf4-d02c27595e38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.198287 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198153    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f14f969b-df8a-4443-951e-4c9318e2362c-cert\") pod \"ingress-canary-glxgf\" (UID: \"f14f969b-df8a-4443-951e-4c9318e2362c\") " pod="openshift-ingress-canary/ingress-canary-glxgf"
Jul 14 15:22:07.198287 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtrm\" (UniqueName: \"kubernetes.io/projected/442ff21f-3879-494d-a460-be0c043a5aa8-kube-api-access-svtrm\") pod \"dns-default-r444t\" (UID: \"442ff21f-3879-494d-a460-be0c043a5aa8\") " pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:07.198287 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/442ff21f-3879-494d-a460-be0c043a5aa8-metrics-tls\") pod \"dns-default-r444t\" (UID: \"442ff21f-3879-494d-a460-be0c043a5aa8\") " pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:07.198287 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a85dd215-4b19-4732-baf4-d02c27595e38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.198287 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a85dd215-4b19-4732-baf4-d02c27595e38-data-volume\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.198287 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/442ff21f-3879-494d-a460-be0c043a5aa8-config-volume\") pod \"dns-default-r444t\" (UID: \"442ff21f-3879-494d-a460-be0c043a5aa8\") " pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:07.198287 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442ff21f-3879-494d-a460-be0c043a5aa8-tmp-dir\") pod \"dns-default-r444t\" (UID: \"442ff21f-3879-494d-a460-be0c043a5aa8\") " pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:07.198685 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wct5q\" (UniqueName: \"kubernetes.io/projected/f14f969b-df8a-4443-951e-4c9318e2362c-kube-api-access-wct5q\") pod \"ingress-canary-glxgf\" (UID: \"f14f969b-df8a-4443-951e-4c9318e2362c\") " pod="openshift-ingress-canary/ingress-canary-glxgf"
Jul 14 15:22:07.198685 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a85dd215-4b19-4732-baf4-d02c27595e38-crio-socket\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.198685 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a85dd215-4b19-4732-baf4-d02c27595e38-crio-socket\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.198861 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198840    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a85dd215-4b19-4732-baf4-d02c27595e38-data-volume\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.198915 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442ff21f-3879-494d-a460-be0c043a5aa8-tmp-dir\") pod \"dns-default-r444t\" (UID: \"442ff21f-3879-494d-a460-be0c043a5aa8\") " pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:07.198915 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.198869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a85dd215-4b19-4732-baf4-d02c27595e38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.199071 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.199049    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/442ff21f-3879-494d-a460-be0c043a5aa8-config-volume\") pod \"dns-default-r444t\" (UID: \"442ff21f-3879-494d-a460-be0c043a5aa8\") " pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:07.202212 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.202183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a85dd215-4b19-4732-baf4-d02c27595e38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.202304 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.202190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/442ff21f-3879-494d-a460-be0c043a5aa8-metrics-tls\") pod \"dns-default-r444t\" (UID: \"442ff21f-3879-494d-a460-be0c043a5aa8\") " pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:07.202304 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.202232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f14f969b-df8a-4443-951e-4c9318e2362c-cert\") pod \"ingress-canary-glxgf\" (UID: \"f14f969b-df8a-4443-951e-4c9318e2362c\") " pod="openshift-ingress-canary/ingress-canary-glxgf"
Jul 14 15:22:07.208031 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.208006    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdwxn\" (UniqueName: \"kubernetes.io/projected/a85dd215-4b19-4732-baf4-d02c27595e38-kube-api-access-hdwxn\") pod \"insights-runtime-extractor-vmlvz\" (UID: \"a85dd215-4b19-4732-baf4-d02c27595e38\") " pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.208359 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.208339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtrm\" (UniqueName: \"kubernetes.io/projected/442ff21f-3879-494d-a460-be0c043a5aa8-kube-api-access-svtrm\") pod \"dns-default-r444t\" (UID: \"442ff21f-3879-494d-a460-be0c043a5aa8\") " pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:07.208598 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.208584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wct5q\" (UniqueName: \"kubernetes.io/projected/f14f969b-df8a-4443-951e-4c9318e2362c-kube-api-access-wct5q\") pod \"ingress-canary-glxgf\" (UID: \"f14f969b-df8a-4443-951e-4c9318e2362c\") " pod="openshift-ingress-canary/ingress-canary-glxgf"
Jul 14 15:22:07.210393 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.210375    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:22:07.210484 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.210375    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:22:07.210547 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.210381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:22:07.213690 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.213669    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vcjnm\""
Jul 14 15:22:07.213880 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.213867    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 15:22:07.213953 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.213920    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zdwzw\""
Jul 14 15:22:07.214037 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.213990    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 15:22:07.214104 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.214042    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 15:22:07.214163 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.214139    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 15:22:07.344472 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.344395    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-glxgf"
Jul 14 15:22:07.366612 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.366585    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:07.390789 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.390763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vmlvz"
Jul 14 15:22:07.552938 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.552894    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-glxgf"]
Jul 14 15:22:07.557939 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:07.557910    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14f969b_df8a_4443_951e_4c9318e2362c.slice/crio-b81401e25e17d64b766e34cef75c31c71533fd4f7ec7c5e37c8978bf7f658bab WatchSource:0}: Error finding container b81401e25e17d64b766e34cef75c31c71533fd4f7ec7c5e37c8978bf7f658bab: Status 404 returned error can't find the container with id b81401e25e17d64b766e34cef75c31c71533fd4f7ec7c5e37c8978bf7f658bab
Jul 14 15:22:07.564474 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.564451    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r444t"]
Jul 14 15:22:07.571736 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:07.571714    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442ff21f_3879_494d_a460_be0c043a5aa8.slice/crio-f132214acdc22ec17a38afeadf657e99f512a2b4c8885423c1665126cb45c915 WatchSource:0}: Error finding container f132214acdc22ec17a38afeadf657e99f512a2b4c8885423c1665126cb45c915: Status 404 returned error can't find the container with id f132214acdc22ec17a38afeadf657e99f512a2b4c8885423c1665126cb45c915
Jul 14 15:22:07.576557 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:07.576538    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vmlvz"]
Jul 14 15:22:07.579314 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:07.579292    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85dd215_4b19_4732_baf4_d02c27595e38.slice/crio-041b3c3ec0ea519950c68536f838dd5b99a70a0a48e8b876872bc0d2700e4d90 WatchSource:0}: Error finding container 041b3c3ec0ea519950c68536f838dd5b99a70a0a48e8b876872bc0d2700e4d90: Status 404 returned error can't find the container with id 041b3c3ec0ea519950c68536f838dd5b99a70a0a48e8b876872bc0d2700e4d90
Jul 14 15:22:08.461795 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:08.461749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r444t" event={"ID":"442ff21f-3879-494d-a460-be0c043a5aa8","Type":"ContainerStarted","Data":"f132214acdc22ec17a38afeadf657e99f512a2b4c8885423c1665126cb45c915"}
Jul 14 15:22:08.463651 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:08.463454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-glxgf" event={"ID":"f14f969b-df8a-4443-951e-4c9318e2362c","Type":"ContainerStarted","Data":"b81401e25e17d64b766e34cef75c31c71533fd4f7ec7c5e37c8978bf7f658bab"}
Jul 14 15:22:08.465715 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:08.465649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmlvz" event={"ID":"a85dd215-4b19-4732-baf4-d02c27595e38","Type":"ContainerStarted","Data":"041b3c3ec0ea519950c68536f838dd5b99a70a0a48e8b876872bc0d2700e4d90"}
Jul 14 15:22:13.479764 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.479727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-glxgf" event={"ID":"f14f969b-df8a-4443-951e-4c9318e2362c","Type":"ContainerStarted","Data":"5c550e13149a8bba34f3a8159bbcb8796712b6ea68c6575b20b5416a09e6bfb6"}
Jul 14 15:22:13.481282 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.481250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmlvz" event={"ID":"a85dd215-4b19-4732-baf4-d02c27595e38","Type":"ContainerStarted","Data":"5c550eba7d1dca199ed833e486ae261d170b1ddae73546355ffde2a6835dc5d4"}
Jul 14 15:22:13.483095 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.483073    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r444t" event={"ID":"442ff21f-3879-494d-a460-be0c043a5aa8","Type":"ContainerStarted","Data":"4f79f46052e76594ddab3fd194c8966d90b8a2c4c7b5771a74d67c6d8b789e9d"}
Jul 14 15:22:13.483208 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.483109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r444t" event={"ID":"442ff21f-3879-494d-a460-be0c043a5aa8","Type":"ContainerStarted","Data":"2ad96967fccc044006ca3c93ae4ae04de47a5f088ab0a4136da52a656b6a4183"}
Jul 14 15:22:13.483269 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.483214    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:13.501347 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.501321    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58949658f4-rm47p"]
Jul 14 15:22:13.502102 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.502057    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-glxgf" podStartSLOduration=2.168209535 podStartE2EDuration="7.502038997s" totalImagesPullingTime="5.333829462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:13.50012497 +0000 UTC m=+57.841390894" watchObservedRunningTime="2026-07-14 15:22:13.502038997 +0000 UTC m=+57.843304905"
Jul 14 15:22:13.503667 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.503648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.507491 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.507466    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 14 15:22:13.507819 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.507802    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 14 15:22:13.507899 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.507870    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 14 15:22:13.508011 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.507995    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 14 15:22:13.508088 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.508069    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 14 15:22:13.508179 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.508165    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 14 15:22:13.508381 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.508366    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 14 15:22:13.508574 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.508560    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7bbbn\""
Jul 14 15:22:13.516630 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.516603    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58949658f4-rm47p"]
Jul 14 15:22:13.519354 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.519302    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r444t" podStartSLOduration=2.223026902 podStartE2EDuration="7.519280652s" totalImagesPullingTime="5.29625375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:13.518428714 +0000 UTC m=+57.859694636" watchObservedRunningTime="2026-07-14 15:22:13.519280652 +0000 UTC m=+57.860546559"
Jul 14 15:22:13.553264 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.553233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/213912a0-c209-4463-b314-654cb2d17075-console-oauth-config\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.553264 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.553267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-oauth-serving-cert\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.553492 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.553470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-console-config\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.553559 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.553529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-service-ca\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.553598 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.553571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78ms7\" (UniqueName: \"kubernetes.io/projected/213912a0-c209-4463-b314-654cb2d17075-kube-api-access-78ms7\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.553720 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.553701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/213912a0-c209-4463-b314-654cb2d17075-console-serving-cert\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.654907 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.654877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/213912a0-c209-4463-b314-654cb2d17075-console-oauth-config\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.654907 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.654908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-oauth-serving-cert\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.655094 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.654941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-console-config\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.655094 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.654957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-service-ca\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.655094 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.654972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78ms7\" (UniqueName: \"kubernetes.io/projected/213912a0-c209-4463-b314-654cb2d17075-kube-api-access-78ms7\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.655243 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.655103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/213912a0-c209-4463-b314-654cb2d17075-console-serving-cert\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.655728 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.655701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-oauth-serving-cert\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.655837 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.655712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-service-ca\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.655837 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.655769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-console-config\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.657400 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.657380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/213912a0-c209-4463-b314-654cb2d17075-console-oauth-config\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.657400 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.657389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/213912a0-c209-4463-b314-654cb2d17075-console-serving-cert\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.663589 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.663566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78ms7\" (UniqueName: \"kubernetes.io/projected/213912a0-c209-4463-b314-654cb2d17075-kube-api-access-78ms7\") pod \"console-58949658f4-rm47p\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") " pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.815845 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.815762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:13.938203 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:13.938057    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58949658f4-rm47p"]
Jul 14 15:22:13.940706 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:13.940678    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213912a0_c209_4463_b314_654cb2d17075.slice/crio-8816a641f1354ac802b494fb1a4242bcbb98f9859d8cc741a2696530bf95efd2 WatchSource:0}: Error finding container 8816a641f1354ac802b494fb1a4242bcbb98f9859d8cc741a2696530bf95efd2: Status 404 returned error can't find the container with id 8816a641f1354ac802b494fb1a4242bcbb98f9859d8cc741a2696530bf95efd2
Jul 14 15:22:14.174927 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:14.174889    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdprn"]
Jul 14 15:22:14.177388 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:14.177367    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdprn"
Jul 14 15:22:14.180736 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:14.180667    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 14 15:22:14.180736 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:14.180676    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ppg4q\""
Jul 14 15:22:14.187941 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:14.187918    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdprn"]
Jul 14 15:22:14.260895 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:14.260859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/14e8e235-8186-4544-99fa-932691a2df79-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-gdprn\" (UID: \"14e8e235-8186-4544-99fa-932691a2df79\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdprn"
Jul 14 15:22:14.362110 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:14.362080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/14e8e235-8186-4544-99fa-932691a2df79-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-gdprn\" (UID: \"14e8e235-8186-4544-99fa-932691a2df79\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdprn"
Jul 14 15:22:14.362345 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:14.362272    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 14 15:22:14.362410 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:14.362349    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e8e235-8186-4544-99fa-932691a2df79-tls-certificates podName:14e8e235-8186-4544-99fa-932691a2df79 nodeName:}" failed. No retries permitted until 2026-07-14 15:22:14.862328888 +0000 UTC m=+59.203594786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/14e8e235-8186-4544-99fa-932691a2df79-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-gdprn" (UID: "14e8e235-8186-4544-99fa-932691a2df79") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 14 15:22:14.490137 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:14.489999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmlvz" event={"ID":"a85dd215-4b19-4732-baf4-d02c27595e38","Type":"ContainerStarted","Data":"87c526c247d8a8e58b6c7387b97c7035379ade7e3c924b5d0406870e5ccd1c73"}
Jul 14 15:22:14.491421 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:14.491390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58949658f4-rm47p" event={"ID":"213912a0-c209-4463-b314-654cb2d17075","Type":"ContainerStarted","Data":"8816a641f1354ac802b494fb1a4242bcbb98f9859d8cc741a2696530bf95efd2"}
Jul 14 15:22:14.865964 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:14.865871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/14e8e235-8186-4544-99fa-932691a2df79-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-gdprn\" (UID: \"14e8e235-8186-4544-99fa-932691a2df79\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdprn"
Jul 14 15:22:14.871529 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:14.869132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/14e8e235-8186-4544-99fa-932691a2df79-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-gdprn\" (UID: \"14e8e235-8186-4544-99fa-932691a2df79\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdprn"
Jul 14 15:22:15.088404 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:15.088366    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdprn"
Jul 14 15:22:16.303303 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:16.303272    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdprn"]
Jul 14 15:22:17.363171 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:17.363117    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e8e235_8186_4544_99fa_932691a2df79.slice/crio-855b33a73e0fe8e180731841d095dbc5720829a557518328bf676c26bb63d379 WatchSource:0}: Error finding container 855b33a73e0fe8e180731841d095dbc5720829a557518328bf676c26bb63d379: Status 404 returned error can't find the container with id 855b33a73e0fe8e180731841d095dbc5720829a557518328bf676c26bb63d379
Jul 14 15:22:17.416087 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.416063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jfnl2"
Jul 14 15:22:17.502130 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.502092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdprn" event={"ID":"14e8e235-8186-4544-99fa-932691a2df79","Type":"ContainerStarted","Data":"855b33a73e0fe8e180731841d095dbc5720829a557518328bf676c26bb63d379"}
Jul 14 15:22:17.503993 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.503968    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmlvz" event={"ID":"a85dd215-4b19-4732-baf4-d02c27595e38","Type":"ContainerStarted","Data":"98b4b2eb4f5c24690790d06f5aa96c220e7e685319104c2067ddda689fe7d302"}
Jul 14 15:22:17.523441 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.523398    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vmlvz" podStartSLOduration=2.068177164 podStartE2EDuration="10.523382282s" totalImagesPullingTime="8.455205118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:22:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:17.522837874 +0000 UTC m=+61.864103796" watchObservedRunningTime="2026-07-14 15:22:17.523382282 +0000 UTC m=+61.864648190"
Jul 14 15:22:17.886434 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.886397    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-746764f8c4-4p5q2"]
Jul 14 15:22:17.900572 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.899414    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-746764f8c4-4p5q2"]
Jul 14 15:22:17.900572 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.899473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:17.907452 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.907430    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 14 15:22:17.990790 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.990746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-config\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:17.990946 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.990815    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-trusted-ca-bundle\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:17.990946 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.990845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-service-ca\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:17.990946 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.990870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g55c\" (UniqueName: \"kubernetes.io/projected/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-kube-api-access-5g55c\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:17.990946 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.990900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-serving-cert\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:17.990946 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.990923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-oauth-serving-cert\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:17.991276 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:17.991027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-oauth-config\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.091442 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.091397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-config\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.091653 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.091459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-trusted-ca-bundle\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.091653 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.091490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-service-ca\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.091653 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.091527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g55c\" (UniqueName: \"kubernetes.io/projected/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-kube-api-access-5g55c\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.091800 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.091658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-serving-cert\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.091800 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.091699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-oauth-serving-cert\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.091800 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.091776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-oauth-config\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.092234 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.092208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-config\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.092357 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.092329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-service-ca\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.092587 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.092561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-trusted-ca-bundle\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.092957 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.092934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-oauth-serving-cert\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.094681 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.094660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-serving-cert\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.094817 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.094794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-oauth-config\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.103291 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.103247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g55c\" (UniqueName: \"kubernetes.io/projected/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-kube-api-access-5g55c\") pod \"console-746764f8c4-4p5q2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") " pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.211399 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.211313    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:18.345066 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.345034    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-746764f8c4-4p5q2"]
Jul 14 15:22:18.348562 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:18.348529    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4a7344_19e2_4ae6_ac29_cffbed75b5b2.slice/crio-214289cb606cb3b6bfbaf3fe91bcae750e3f7e1a1e251379008943947e60bf13 WatchSource:0}: Error finding container 214289cb606cb3b6bfbaf3fe91bcae750e3f7e1a1e251379008943947e60bf13: Status 404 returned error can't find the container with id 214289cb606cb3b6bfbaf3fe91bcae750e3f7e1a1e251379008943947e60bf13
Jul 14 15:22:18.509476 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.509377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746764f8c4-4p5q2" event={"ID":"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2","Type":"ContainerStarted","Data":"a242ee73a1e4224396c4ee9a8963d5fc218da0d32f78bce024d6080339dc6e8a"}
Jul 14 15:22:18.509476 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.509443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746764f8c4-4p5q2" event={"ID":"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2","Type":"ContainerStarted","Data":"214289cb606cb3b6bfbaf3fe91bcae750e3f7e1a1e251379008943947e60bf13"}
Jul 14 15:22:18.511367 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.511343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58949658f4-rm47p" event={"ID":"213912a0-c209-4463-b314-654cb2d17075","Type":"ContainerStarted","Data":"91e89e77410f7671800f4284124878359bddda5cdb44f500eb3fa9a5fb0864d3"}
Jul 14 15:22:18.528946 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.528889    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-746764f8c4-4p5q2" podStartSLOduration=1.5288720040000001 podStartE2EDuration="1.528872004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:18.528112298 +0000 UTC m=+62.869378228" watchObservedRunningTime="2026-07-14 15:22:18.528872004 +0000 UTC m=+62.870137911"
Jul 14 15:22:18.554703 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:18.554654    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58949658f4-rm47p" podStartSLOduration=1.8557180739999999 podStartE2EDuration="5.554632621s" totalImagesPullingTime="3.698914547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:18.553739171 +0000 UTC m=+62.895005099" watchObservedRunningTime="2026-07-14 15:22:18.554632621 +0000 UTC m=+62.895898529"
Jul 14 15:22:19.515379 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:19.515339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdprn" event={"ID":"14e8e235-8186-4544-99fa-932691a2df79","Type":"ContainerStarted","Data":"16a06ffbaa9be3152bd728e0f0f665ec0315034a4d585df99eb3b4a4fc3ab3b6"}
Jul 14 15:22:19.515862 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:19.515780    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdprn"
Jul 14 15:22:19.520269 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:19.520246    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdprn"
Jul 14 15:22:19.533343 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:19.533296    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdprn" podStartSLOduration=4.21769189 podStartE2EDuration="5.533283348s" totalImagesPullingTime="1.315591458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:19.532012972 +0000 UTC m=+63.873278878" watchObservedRunningTime="2026-07-14 15:22:19.533283348 +0000 UTC m=+63.874549255"
Jul 14 15:22:20.286116 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.286085    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-f7bkz"]
Jul 14 15:22:20.290102 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.288944    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz"
Jul 14 15:22:20.292539 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.292517    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 14 15:22:20.292539 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.292528    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 14 15:22:20.292707 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.292566    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xch29\""
Jul 14 15:22:20.294330 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.294315    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 14 15:22:20.294388 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.294373    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 14 15:22:20.294425 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.294409    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 14 15:22:20.300050 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.300030    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-f7bkz"]
Jul 14 15:22:20.410635 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.410601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec57c675-9574-465f-9123-e5c3bda94761-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-f7bkz\" (UID: \"ec57c675-9574-465f-9123-e5c3bda94761\") " pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz"
Jul 14 15:22:20.410786 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.410654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec57c675-9574-465f-9123-e5c3bda94761-metrics-client-ca\") pod \"prometheus-operator-64785b6675-f7bkz\" (UID: \"ec57c675-9574-465f-9123-e5c3bda94761\") " pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz"
Jul 14 15:22:20.410786 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.410691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec57c675-9574-465f-9123-e5c3bda94761-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-f7bkz\" (UID: \"ec57c675-9574-465f-9123-e5c3bda94761\") " pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz"
Jul 14 15:22:20.410786 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.410724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kttqn\" (UniqueName: \"kubernetes.io/projected/ec57c675-9574-465f-9123-e5c3bda94761-kube-api-access-kttqn\") pod \"prometheus-operator-64785b6675-f7bkz\" (UID: \"ec57c675-9574-465f-9123-e5c3bda94761\") " pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz"
Jul 14 15:22:20.511285 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.511255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec57c675-9574-465f-9123-e5c3bda94761-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-f7bkz\" (UID: \"ec57c675-9574-465f-9123-e5c3bda94761\") " pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz"
Jul 14 15:22:20.511489 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.511324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec57c675-9574-465f-9123-e5c3bda94761-metrics-client-ca\") pod \"prometheus-operator-64785b6675-f7bkz\" (UID: \"ec57c675-9574-465f-9123-e5c3bda94761\") " pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz"
Jul 14 15:22:20.511489 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.511380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec57c675-9574-465f-9123-e5c3bda94761-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-f7bkz\" (UID: \"ec57c675-9574-465f-9123-e5c3bda94761\") " pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz"
Jul 14 15:22:20.511489 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.511432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kttqn\" (UniqueName: \"kubernetes.io/projected/ec57c675-9574-465f-9123-e5c3bda94761-kube-api-access-kttqn\") pod \"prometheus-operator-64785b6675-f7bkz\" (UID: \"ec57c675-9574-465f-9123-e5c3bda94761\") " pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz"
Jul 14 15:22:20.512108 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.512024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec57c675-9574-465f-9123-e5c3bda94761-metrics-client-ca\") pod \"prometheus-operator-64785b6675-f7bkz\" (UID: \"ec57c675-9574-465f-9123-e5c3bda94761\") " pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz"
Jul 14 15:22:20.513755 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.513736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec57c675-9574-465f-9123-e5c3bda94761-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-f7bkz\" (UID: \"ec57c675-9574-465f-9123-e5c3bda94761\") " pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz"
Jul 14 15:22:20.513850 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.513803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec57c675-9574-465f-9123-e5c3bda94761-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-f7bkz\" (UID: \"ec57c675-9574-465f-9123-e5c3bda94761\") " pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz"
Jul 14 15:22:20.521052 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.521033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttqn\" (UniqueName: \"kubernetes.io/projected/ec57c675-9574-465f-9123-e5c3bda94761-kube-api-access-kttqn\") pod \"prometheus-operator-64785b6675-f7bkz\" (UID: \"ec57c675-9574-465f-9123-e5c3bda94761\") " pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz"
Jul 14 15:22:20.599058 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.598971    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz"
Jul 14 15:22:20.723283 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:20.723252    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-f7bkz"]
Jul 14 15:22:20.725933 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:20.725904    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec57c675_9574_465f_9123_e5c3bda94761.slice/crio-6cf12cc5f3502ff64a1bae0c099d29bd5607600e1fb591123b58bfa4e2195ae4 WatchSource:0}: Error finding container 6cf12cc5f3502ff64a1bae0c099d29bd5607600e1fb591123b58bfa4e2195ae4: Status 404 returned error can't find the container with id 6cf12cc5f3502ff64a1bae0c099d29bd5607600e1fb591123b58bfa4e2195ae4
Jul 14 15:22:21.523093 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:21.523060    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz" event={"ID":"ec57c675-9574-465f-9123-e5c3bda94761","Type":"ContainerStarted","Data":"6cf12cc5f3502ff64a1bae0c099d29bd5607600e1fb591123b58bfa4e2195ae4"}
Jul 14 15:22:21.922903 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:21.922861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:22:21.923083 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:21.922950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqm7w\" (UniqueName: \"kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w\") pod \"network-check-target-w82g9\" (UID: \"bf1192e2-ad6e-4efc-964e-3c31e509c8e6\") " pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:22:21.926147 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:21.926119    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 15:22:21.926283 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:21.926119    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 15:22:21.936245 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:21.936221    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 15:22:21.936348 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:21.936258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ae120a-8040-483a-8da6-090ef1e80265-original-pull-secret\") pod \"global-pull-secret-syncer-w8dm9\" (UID: \"d3ae120a-8040-483a-8da6-090ef1e80265\") " pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:22:21.947394 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:21.947371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqm7w\" (UniqueName: \"kubernetes.io/projected/bf1192e2-ad6e-4efc-964e-3c31e509c8e6-kube-api-access-bqm7w\") pod \"network-check-target-w82g9\" (UID: \"bf1192e2-ad6e-4efc-964e-3c31e509c8e6\") " pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:22:22.023791 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.023766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs\") pod \"network-metrics-daemon-frtx9\" (UID: \"b42135f6-4895-4ca1-8086-1ae81b8006c8\") " pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:22:22.026599 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.026579    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 15:22:22.036517 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.036479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42135f6-4895-4ca1-8086-1ae81b8006c8-metrics-certs\") pod \"network-metrics-daemon-frtx9\" (UID: \"b42135f6-4895-4ca1-8086-1ae81b8006c8\") " pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:22:22.226388 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.226306    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vcjnm\""
Jul 14 15:22:22.232037 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.232021    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zdwzw\""
Jul 14 15:22:22.234087 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.234067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frtx9"
Jul 14 15:22:22.234183 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.234170    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8dm9"
Jul 14 15:22:22.239883 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.239868    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:22:22.382617 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.382576    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-frtx9"]
Jul 14 15:22:22.386837 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:22.386813    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42135f6_4895_4ca1_8086_1ae81b8006c8.slice/crio-81777a8c62499f1222a4640bced95ed47fb329d0fdc8f1d7b4d837ec05067ff9 WatchSource:0}: Error finding container 81777a8c62499f1222a4640bced95ed47fb329d0fdc8f1d7b4d837ec05067ff9: Status 404 returned error can't find the container with id 81777a8c62499f1222a4640bced95ed47fb329d0fdc8f1d7b4d837ec05067ff9
Jul 14 15:22:22.401406 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.401383    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w82g9"]
Jul 14 15:22:22.404067 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:22.404043    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1192e2_ad6e_4efc_964e_3c31e509c8e6.slice/crio-9d5556e63373f8e2be6a5d8a1e930a2d25787a1fa8b179adb696faca3ff43ecf WatchSource:0}: Error finding container 9d5556e63373f8e2be6a5d8a1e930a2d25787a1fa8b179adb696faca3ff43ecf: Status 404 returned error can't find the container with id 9d5556e63373f8e2be6a5d8a1e930a2d25787a1fa8b179adb696faca3ff43ecf
Jul 14 15:22:22.417784 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.417755    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w8dm9"]
Jul 14 15:22:22.420239 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:22.420218    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ae120a_8040_483a_8da6_090ef1e80265.slice/crio-9049ec942591800ae3646a0d4cb934df24a3a0ef8ba8b4f2b8b7bbdf02038ead WatchSource:0}: Error finding container 9049ec942591800ae3646a0d4cb934df24a3a0ef8ba8b4f2b8b7bbdf02038ead: Status 404 returned error can't find the container with id 9049ec942591800ae3646a0d4cb934df24a3a0ef8ba8b4f2b8b7bbdf02038ead
Jul 14 15:22:22.528961 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.528883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w8dm9" event={"ID":"d3ae120a-8040-483a-8da6-090ef1e80265","Type":"ContainerStarted","Data":"9049ec942591800ae3646a0d4cb934df24a3a0ef8ba8b4f2b8b7bbdf02038ead"}
Jul 14 15:22:22.530757 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.530732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz" event={"ID":"ec57c675-9574-465f-9123-e5c3bda94761","Type":"ContainerStarted","Data":"39a18fcb1151e0c6367954ad1cf4cc9162732d41efe1afa0be964ec3d2dd261d"}
Jul 14 15:22:22.530927 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.530778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz" event={"ID":"ec57c675-9574-465f-9123-e5c3bda94761","Type":"ContainerStarted","Data":"26d6be5be2da82807e552cb49abf487b44b184fb4ddbe1a3a6aef8bf30ed20f5"}
Jul 14 15:22:22.532077 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.532043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-frtx9" event={"ID":"b42135f6-4895-4ca1-8086-1ae81b8006c8","Type":"ContainerStarted","Data":"81777a8c62499f1222a4640bced95ed47fb329d0fdc8f1d7b4d837ec05067ff9"}
Jul 14 15:22:22.533100 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.533080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w82g9" event={"ID":"bf1192e2-ad6e-4efc-964e-3c31e509c8e6","Type":"ContainerStarted","Data":"9d5556e63373f8e2be6a5d8a1e930a2d25787a1fa8b179adb696faca3ff43ecf"}
Jul 14 15:22:22.549564 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:22.549524    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-f7bkz" podStartSLOduration=1.270824484 podStartE2EDuration="2.549485695s" totalImagesPullingTime="1.278661211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:22.548600442 +0000 UTC m=+66.889866368" watchObservedRunningTime="2026-07-14 15:22:22.549485695 +0000 UTC m=+66.890751603"
Jul 14 15:22:23.495836 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:23.495802    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r444t"
Jul 14 15:22:23.816710 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:23.816614    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:23.817168 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:23.816750    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:23.822906 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:23.822881    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:24.546138 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.545700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-frtx9" event={"ID":"b42135f6-4895-4ca1-8086-1ae81b8006c8","Type":"ContainerStarted","Data":"76a68d22ec725d105cb2f836a240d6f396ab238d8b4a0d272e0fbb74d7f4fa55"}
Jul 14 15:22:24.546138 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.545758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-frtx9" event={"ID":"b42135f6-4895-4ca1-8086-1ae81b8006c8","Type":"ContainerStarted","Data":"53c8a003b80d5b03d66b57c3e5cf199037e648b454c10a8d29cf9386e1a9a7f0"}
Jul 14 15:22:24.550595 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.550564    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:24.573370 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.573302    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-frtx9" podStartSLOduration=66.491334054 podStartE2EDuration="1m7.573280422s" totalImagesPullingTime="1.081946368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:24.572365731 +0000 UTC m=+68.913631656" watchObservedRunningTime="2026-07-14 15:22:24.573280422 +0000 UTC m=+68.914546331"
Jul 14 15:22:24.695861 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.695822    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"]
Jul 14 15:22:24.728601 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.728562    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"]
Jul 14 15:22:24.728784 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.728703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:24.731948 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.731916    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-87rgz\""
Jul 14 15:22:24.732125 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.732104    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 14 15:22:24.732240 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.732140    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 14 15:22:24.756136 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.756103    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"]
Jul 14 15:22:24.772415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.772365    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7k52l"]
Jul 14 15:22:24.772599 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.772547    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.778534 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.776084    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 14 15:22:24.778534 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.776143    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cfnqb\""
Jul 14 15:22:24.778534 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.776209    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 14 15:22:24.778534 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.776084    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 14 15:22:24.794963 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.794930    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"]
Jul 14 15:22:24.795102 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.795084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.798172 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.798113    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 14 15:22:24.798344 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.798328    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 14 15:22:24.798586 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.798567    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 14 15:22:24.798784 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.798767    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k74s7\""
Jul 14 15:22:24.847893 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.847854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49e14489-568d-455c-97a2-241625d0155a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.847906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-w4r2z\" (UID: \"95c86f2f-cfd8-4af0-ab06-4812eb0e8886\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.847942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-w4r2z\" (UID: \"95c86f2f-cfd8-4af0-ab06-4812eb0e8886\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.847968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-textfile\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.847996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.848037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-root\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.848066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-metrics-client-ca\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.848090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkbnt\" (UniqueName: \"kubernetes.io/projected/49e14489-568d-455c-97a2-241625d0155a-kube-api-access-tkbnt\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.848119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbxjj\" (UniqueName: \"kubernetes.io/projected/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-kube-api-access-nbxjj\") pod \"openshift-state-metrics-84bd68b896-w4r2z\" (UID: \"95c86f2f-cfd8-4af0-ab06-4812eb0e8886\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.848160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/49e14489-568d-455c-97a2-241625d0155a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.848185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/49e14489-568d-455c-97a2-241625d0155a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.848211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e14489-568d-455c-97a2-241625d0155a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.848241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-sys\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.848265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtsmd\" (UniqueName: \"kubernetes.io/projected/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-kube-api-access-jtsmd\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.848294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-accelerators-collector-config\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.848356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.848326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/49e14489-568d-455c-97a2-241625d0155a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.849047 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.848348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-tls\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.849047 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.848370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-wtmp\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.849047 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.848410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-w4r2z\" (UID: \"95c86f2f-cfd8-4af0-ab06-4812eb0e8886\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:24.949260 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-sys\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.949451 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtsmd\" (UniqueName: \"kubernetes.io/projected/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-kube-api-access-jtsmd\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.949451 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-accelerators-collector-config\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.949451 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-sys\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.949451 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/49e14489-568d-455c-97a2-241625d0155a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.949451 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-tls\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.949451 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-wtmp\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.949451 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-w4r2z\" (UID: \"95c86f2f-cfd8-4af0-ab06-4812eb0e8886\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:24.949451 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49e14489-568d-455c-97a2-241625d0155a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.949451 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-w4r2z\" (UID: \"95c86f2f-cfd8-4af0-ab06-4812eb0e8886\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:24.949934 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-w4r2z\" (UID: \"95c86f2f-cfd8-4af0-ab06-4812eb0e8886\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:24.949934 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-textfile\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.949934 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.949934 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-root\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.949934 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-metrics-client-ca\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.949934 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkbnt\" (UniqueName: \"kubernetes.io/projected/49e14489-568d-455c-97a2-241625d0155a-kube-api-access-tkbnt\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.949934 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbxjj\" (UniqueName: \"kubernetes.io/projected/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-kube-api-access-nbxjj\") pod \"openshift-state-metrics-84bd68b896-w4r2z\" (UID: \"95c86f2f-cfd8-4af0-ab06-4812eb0e8886\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:24.949934 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/49e14489-568d-455c-97a2-241625d0155a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.949934 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/49e14489-568d-455c-97a2-241625d0155a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.949934 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e14489-568d-455c-97a2-241625d0155a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.950394 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.949979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-accelerators-collector-config\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.950394 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.950129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-wtmp\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.950394 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.950376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-textfile\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.950572 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.950469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-metrics-client-ca\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.950634 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:24.950581    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 14 15:22:24.950691 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.950628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e14489-568d-455c-97a2-241625d0155a-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.950691 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:24.950676    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-tls podName:ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2 nodeName:}" failed. No retries permitted until 2026-07-14 15:22:25.450650582 +0000 UTC m=+69.791916475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-tls") pod "node-exporter-7k52l" (UID: "ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2") : secret "node-exporter-tls" not found
Jul 14 15:22:24.951153 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.950935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-root\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.951153 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.951007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-w4r2z\" (UID: \"95c86f2f-cfd8-4af0-ab06-4812eb0e8886\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:24.951153 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:24.951026    2563 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 14 15:22:24.951153 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:24.951060    2563 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 14 15:22:24.951153 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:24.951068    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49e14489-568d-455c-97a2-241625d0155a-kube-state-metrics-tls podName:49e14489-568d-455c-97a2-241625d0155a nodeName:}" failed. No retries permitted until 2026-07-14 15:22:25.451053538 +0000 UTC m=+69.792319423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/49e14489-568d-455c-97a2-241625d0155a-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-lc7kr" (UID: "49e14489-568d-455c-97a2-241625d0155a") : secret "kube-state-metrics-tls" not found
Jul 14 15:22:24.951970 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:24.951947    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-openshift-state-metrics-tls podName:95c86f2f-cfd8-4af0-ab06-4812eb0e8886 nodeName:}" failed. No retries permitted until 2026-07-14 15:22:25.451918149 +0000 UTC m=+69.793184045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-w4r2z" (UID: "95c86f2f-cfd8-4af0-ab06-4812eb0e8886") : secret "openshift-state-metrics-tls" not found
Jul 14 15:22:24.952249 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.952227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/49e14489-568d-455c-97a2-241625d0155a-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.952615 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.952582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/49e14489-568d-455c-97a2-241625d0155a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.953854 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.953830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49e14489-568d-455c-97a2-241625d0155a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:24.954644 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.954623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-w4r2z\" (UID: \"95c86f2f-cfd8-4af0-ab06-4812eb0e8886\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:24.956544 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.954818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.960718 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.960670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtsmd\" (UniqueName: \"kubernetes.io/projected/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-kube-api-access-jtsmd\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:24.960821 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.960733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbxjj\" (UniqueName: \"kubernetes.io/projected/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-kube-api-access-nbxjj\") pod \"openshift-state-metrics-84bd68b896-w4r2z\" (UID: \"95c86f2f-cfd8-4af0-ab06-4812eb0e8886\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:24.960920 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:24.960899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkbnt\" (UniqueName: \"kubernetes.io/projected/49e14489-568d-455c-97a2-241625d0155a-kube-api-access-tkbnt\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:25.453813 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.453767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-w4r2z\" (UID: \"95c86f2f-cfd8-4af0-ab06-4812eb0e8886\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:25.453993 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.453875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/49e14489-568d-455c-97a2-241625d0155a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:25.453993 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.453926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-tls\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:25.456563 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.456534    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2-node-exporter-tls\") pod \"node-exporter-7k52l\" (UID: \"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2\") " pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:25.456685 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.456650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c86f2f-cfd8-4af0-ab06-4812eb0e8886-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-w4r2z\" (UID: \"95c86f2f-cfd8-4af0-ab06-4812eb0e8886\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:25.456800 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.456777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/49e14489-568d-455c-97a2-241625d0155a-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-lc7kr\" (UID: \"49e14489-568d-455c-97a2-241625d0155a\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:25.641390 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.641350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"
Jul 14 15:22:25.684636 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.684601    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"
Jul 14 15:22:25.706585 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.706484    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7k52l"
Jul 14 15:22:25.787611 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.787580    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 15:22:25.790229 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.790206    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.793494 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.793400    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 14 15:22:25.793494 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.793484    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 14 15:22:25.793728 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.793674    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 14 15:22:25.793851 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.793831    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 14 15:22:25.794050 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.793970    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 14 15:22:25.794349 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.794327    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-btkbr\""
Jul 14 15:22:25.794446 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.794354    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 14 15:22:25.794533 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.794490    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 14 15:22:25.794596 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.794565    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 14 15:22:25.794596 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.794588    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 14 15:22:25.806818 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.806794    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 15:22:25.858183 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.858142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.858683 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.858204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.858683 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.858268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-config-volume\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.858683 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.858319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.858683 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.858359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63ed7786-f31d-4538-a5d1-29b328cf5c23-tls-assets\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.858683 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.858405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63ed7786-f31d-4538-a5d1-29b328cf5c23-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.858683 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.858452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-web-config\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.858683 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.858482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63ed7786-f31d-4538-a5d1-29b328cf5c23-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.858683 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.858536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.858683 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.858568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63ed7786-f31d-4538-a5d1-29b328cf5c23-config-out\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.858683 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.858589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.858683 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.858651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5gxz\" (UniqueName: \"kubernetes.io/projected/63ed7786-f31d-4538-a5d1-29b328cf5c23-kube-api-access-f5gxz\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.858683 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.858680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63ed7786-f31d-4538-a5d1-29b328cf5c23-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.959120 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.959029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63ed7786-f31d-4538-a5d1-29b328cf5c23-tls-assets\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.959120 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.959078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63ed7786-f31d-4538-a5d1-29b328cf5c23-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.959361 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.959121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-web-config\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.959361 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.959146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63ed7786-f31d-4538-a5d1-29b328cf5c23-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.959361 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.959217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.959361 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.959252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63ed7786-f31d-4538-a5d1-29b328cf5c23-config-out\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.959361 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.959269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.959361 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.959302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5gxz\" (UniqueName: \"kubernetes.io/projected/63ed7786-f31d-4538-a5d1-29b328cf5c23-kube-api-access-f5gxz\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.959361 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.959331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63ed7786-f31d-4538-a5d1-29b328cf5c23-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.959731 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.959376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.959731 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.959418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.959731 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.959438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-config-volume\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.959731 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.959465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.959731 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.959641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63ed7786-f31d-4538-a5d1-29b328cf5c23-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.961082 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.961049    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63ed7786-f31d-4538-a5d1-29b328cf5c23-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.961373 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.961350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63ed7786-f31d-4538-a5d1-29b328cf5c23-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.962393 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.962368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63ed7786-f31d-4538-a5d1-29b328cf5c23-config-out\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.962799 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.962750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63ed7786-f31d-4538-a5d1-29b328cf5c23-tls-assets\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.963524 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.963478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.963699 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.963676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-web-config\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.964181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.964159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.964639 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.964615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.964741 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.964715    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.965121 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.965100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-config-volume\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.965901 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.965874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63ed7786-f31d-4538-a5d1-29b328cf5c23-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:25.968766 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:25.968743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5gxz\" (UniqueName: \"kubernetes.io/projected/63ed7786-f31d-4538-a5d1-29b328cf5c23-kube-api-access-f5gxz\") pod \"alertmanager-main-0\" (UID: \"63ed7786-f31d-4538-a5d1-29b328cf5c23\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:26.102830 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:26.102794    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 14 15:22:27.058588 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:27.058207    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac00d550_f9d9_413d_9cc4_3d6afeb9f9b2.slice/crio-861acce0524de91e1fe3e3ba06b42b821f40d3bb6c22073e608f4249b746d51a WatchSource:0}: Error finding container 861acce0524de91e1fe3e3ba06b42b821f40d3bb6c22073e608f4249b746d51a: Status 404 returned error can't find the container with id 861acce0524de91e1fe3e3ba06b42b821f40d3bb6c22073e608f4249b746d51a
Jul 14 15:22:27.252435 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.252395    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z"]
Jul 14 15:22:27.254837 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:27.254796    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c86f2f_cfd8_4af0_ab06_4812eb0e8886.slice/crio-02f3566deb5ee794a87c6633804c9b65f9a3ae0eca7094fec1b2fa84dbc03ab7 WatchSource:0}: Error finding container 02f3566deb5ee794a87c6633804c9b65f9a3ae0eca7094fec1b2fa84dbc03ab7: Status 404 returned error can't find the container with id 02f3566deb5ee794a87c6633804c9b65f9a3ae0eca7094fec1b2fa84dbc03ab7
Jul 14 15:22:27.474650 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.474621    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-lc7kr"]
Jul 14 15:22:27.476578 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:27.476541    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e14489_568d_455c_97a2_241625d0155a.slice/crio-a9bcbe7539b55b5ed2b57bd90c453313e28e488a5af91bace5a96dce17394709 WatchSource:0}: Error finding container a9bcbe7539b55b5ed2b57bd90c453313e28e488a5af91bace5a96dce17394709: Status 404 returned error can't find the container with id a9bcbe7539b55b5ed2b57bd90c453313e28e488a5af91bace5a96dce17394709
Jul 14 15:22:27.485144 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.485117    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 15:22:27.487156 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:27.487124    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ed7786_f31d_4538_a5d1_29b328cf5c23.slice/crio-980279f3658e69e744bc2c20e8732a241730b180d6c7d2ffba4d33ddbdd3fb54 WatchSource:0}: Error finding container 980279f3658e69e744bc2c20e8732a241730b180d6c7d2ffba4d33ddbdd3fb54: Status 404 returned error can't find the container with id 980279f3658e69e744bc2c20e8732a241730b180d6c7d2ffba4d33ddbdd3fb54
Jul 14 15:22:27.558030 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.557983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w8dm9" event={"ID":"d3ae120a-8040-483a-8da6-090ef1e80265","Type":"ContainerStarted","Data":"21069ae079436cb864a8104d6483e2c21d92598f270feb9b67cba07a09069c0c"}
Jul 14 15:22:27.559512 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.559476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr" event={"ID":"49e14489-568d-455c-97a2-241625d0155a","Type":"ContainerStarted","Data":"a9bcbe7539b55b5ed2b57bd90c453313e28e488a5af91bace5a96dce17394709"}
Jul 14 15:22:27.561034 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.561003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63ed7786-f31d-4538-a5d1-29b328cf5c23","Type":"ContainerStarted","Data":"980279f3658e69e744bc2c20e8732a241730b180d6c7d2ffba4d33ddbdd3fb54"}
Jul 14 15:22:27.563826 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.563751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z" event={"ID":"95c86f2f-cfd8-4af0-ab06-4812eb0e8886","Type":"ContainerStarted","Data":"fc6d0cf5d802b255a670cb3bbfa9ae9f67f543ea6393d3f3ee5f42c03a337fb8"}
Jul 14 15:22:27.563826 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.563816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z" event={"ID":"95c86f2f-cfd8-4af0-ab06-4812eb0e8886","Type":"ContainerStarted","Data":"4ffd09f1b594ff36f121bb898128c8b47b2e015c881d6f83736cb4e19f5fd176"}
Jul 14 15:22:27.563970 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.563846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z" event={"ID":"95c86f2f-cfd8-4af0-ab06-4812eb0e8886","Type":"ContainerStarted","Data":"02f3566deb5ee794a87c6633804c9b65f9a3ae0eca7094fec1b2fa84dbc03ab7"}
Jul 14 15:22:27.565024 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.565004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7k52l" event={"ID":"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2","Type":"ContainerStarted","Data":"861acce0524de91e1fe3e3ba06b42b821f40d3bb6c22073e608f4249b746d51a"}
Jul 14 15:22:27.566617 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.566594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w82g9" event={"ID":"bf1192e2-ad6e-4efc-964e-3c31e509c8e6","Type":"ContainerStarted","Data":"ed4f803934caf8286af73f3f37335f3c1e4372162ac5d5ff60218d63ab25e3d7"}
Jul 14 15:22:27.566760 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.566739    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:22:27.574660 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.574613    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w8dm9" podStartSLOduration=65.928868512 podStartE2EDuration="1m10.574596833s" totalImagesPullingTime="4.645728321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:27.573609433 +0000 UTC m=+71.914875378" watchObservedRunningTime="2026-07-14 15:22:27.574596833 +0000 UTC m=+71.915862717"
Jul 14 15:22:27.592622 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.592565    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w82g9" podStartSLOduration=66.941200881 podStartE2EDuration="1m11.592547455s" totalImagesPullingTime="4.651346574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:27.592207273 +0000 UTC m=+71.933473202" watchObservedRunningTime="2026-07-14 15:22:27.592547455 +0000 UTC m=+71.933813361"
Jul 14 15:22:27.661104 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.661066    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-69748d48dc-wspq7"]
Jul 14 15:22:27.664516 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.664479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.667175 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.667155    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 14 15:22:27.667297 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.667162    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 14 15:22:27.667446 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.667413    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 14 15:22:27.667591 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.667559    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-br27g\""
Jul 14 15:22:27.667591 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.667579    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 14 15:22:27.667707 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.667617    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 14 15:22:27.667707 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.667681    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2ppvug5c3lqd4\""
Jul 14 15:22:27.680972 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.680948    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69748d48dc-wspq7"]
Jul 14 15:22:27.773730 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.773695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.773882 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.773749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-tls\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.773882 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.773798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/998a59ad-a466-47b3-a115-50318de410a3-metrics-client-ca\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.774000 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.773885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.774000 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.773909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnc4s\" (UniqueName: \"kubernetes.io/projected/998a59ad-a466-47b3-a115-50318de410a3-kube-api-access-tnc4s\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.774000 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.773959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.774110 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.774010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.774110 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.774038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-grpc-tls\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.875171 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.875089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.875171 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.875133    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-grpc-tls\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.875414 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.875186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.875414 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.875234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-tls\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.875414 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.875267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/998a59ad-a466-47b3-a115-50318de410a3-metrics-client-ca\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.875414 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.875291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.875414 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.875314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnc4s\" (UniqueName: \"kubernetes.io/projected/998a59ad-a466-47b3-a115-50318de410a3-kube-api-access-tnc4s\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.875414 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.875345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.876154 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.876105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/998a59ad-a466-47b3-a115-50318de410a3-metrics-client-ca\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.878395 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.878351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-grpc-tls\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.878395 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.878368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.878618 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.878368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.878618 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.878445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.878618 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.878539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-tls\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.878847 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.878830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/998a59ad-a466-47b3-a115-50318de410a3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.884453 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.884426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnc4s\" (UniqueName: \"kubernetes.io/projected/998a59ad-a466-47b3-a115-50318de410a3-kube-api-access-tnc4s\") pod \"thanos-querier-69748d48dc-wspq7\" (UID: \"998a59ad-a466-47b3-a115-50318de410a3\") " pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:27.975210 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:27.975180    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:28.129268 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.129193    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69748d48dc-wspq7"]
Jul 14 15:22:28.133068 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:28.133038    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998a59ad_a466_47b3_a115_50318de410a3.slice/crio-506ef8ec3808cb4c58c30da417718dea789f586ae6b0f460fa7244a3559becea WatchSource:0}: Error finding container 506ef8ec3808cb4c58c30da417718dea789f586ae6b0f460fa7244a3559becea: Status 404 returned error can't find the container with id 506ef8ec3808cb4c58c30da417718dea789f586ae6b0f460fa7244a3559becea
Jul 14 15:22:28.218082 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.217849    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:28.218082 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.217905    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:28.218717 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.218548    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:28.573102 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.573062    2563 generic.go:356] "Generic (PLEG): container finished" podID="ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2" containerID="c59f82f5896931a024505d2301ed1399c6390accbee28d65d529013149a83eb9" exitCode=0
Jul 14 15:22:28.573288 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.573150    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7k52l" event={"ID":"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2","Type":"ContainerDied","Data":"c59f82f5896931a024505d2301ed1399c6390accbee28d65d529013149a83eb9"}
Jul 14 15:22:28.574653 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.574626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7" event={"ID":"998a59ad-a466-47b3-a115-50318de410a3","Type":"ContainerStarted","Data":"506ef8ec3808cb4c58c30da417718dea789f586ae6b0f460fa7244a3559becea"}
Jul 14 15:22:28.579298 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.579275    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:22:28.640991 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.640956    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58949658f4-rm47p"]
Jul 14 15:22:28.989301 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.989268    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-59799b7f6c-8zgxl"]
Jul 14 15:22:28.991449 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.991426    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:28.994233 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.994212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 14 15:22:28.995600 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.995568    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dlsormekqqvfo\""
Jul 14 15:22:28.995600 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.995581    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 14 15:22:28.995600 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.995585    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 14 15:22:28.995819 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.995595    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 14 15:22:28.995819 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:28.995639    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4mmfj\""
Jul 14 15:22:29.013144 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.013116    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-59799b7f6c-8zgxl"]
Jul 14 15:22:29.086557 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.086522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d05e0184-6720-4dd3-a436-d9033446f7ac-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.086735 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.086612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d05e0184-6720-4dd3-a436-d9033446f7ac-secret-metrics-server-tls\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.086735 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.086652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d05e0184-6720-4dd3-a436-d9033446f7ac-audit-log\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.086735 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.086681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d05e0184-6720-4dd3-a436-d9033446f7ac-metrics-server-audit-profiles\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.086735 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.086730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d05e0184-6720-4dd3-a436-d9033446f7ac-client-ca-bundle\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.086952 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.086752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2twnd\" (UniqueName: \"kubernetes.io/projected/d05e0184-6720-4dd3-a436-d9033446f7ac-kube-api-access-2twnd\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.086952 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.086830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d05e0184-6720-4dd3-a436-d9033446f7ac-secret-metrics-server-client-certs\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.187652 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.187616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d05e0184-6720-4dd3-a436-d9033446f7ac-client-ca-bundle\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.187652 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.187658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2twnd\" (UniqueName: \"kubernetes.io/projected/d05e0184-6720-4dd3-a436-d9033446f7ac-kube-api-access-2twnd\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.188231 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.187695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d05e0184-6720-4dd3-a436-d9033446f7ac-secret-metrics-server-client-certs\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.188231 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.187749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d05e0184-6720-4dd3-a436-d9033446f7ac-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.188231 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.187794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d05e0184-6720-4dd3-a436-d9033446f7ac-secret-metrics-server-tls\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.188231 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.187919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d05e0184-6720-4dd3-a436-d9033446f7ac-audit-log\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.188231 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.187957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d05e0184-6720-4dd3-a436-d9033446f7ac-metrics-server-audit-profiles\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.189011 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.188979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d05e0184-6720-4dd3-a436-d9033446f7ac-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.189335 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.189311    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d05e0184-6720-4dd3-a436-d9033446f7ac-audit-log\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.189881 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.189861    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d05e0184-6720-4dd3-a436-d9033446f7ac-metrics-server-audit-profiles\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.190878 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.190856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d05e0184-6720-4dd3-a436-d9033446f7ac-secret-metrics-server-tls\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.190878 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.190867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d05e0184-6720-4dd3-a436-d9033446f7ac-client-ca-bundle\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.191630 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.191605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d05e0184-6720-4dd3-a436-d9033446f7ac-secret-metrics-server-client-certs\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.197677 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.197652    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2twnd\" (UniqueName: \"kubernetes.io/projected/d05e0184-6720-4dd3-a436-d9033446f7ac-kube-api-access-2twnd\") pod \"metrics-server-59799b7f6c-8zgxl\" (UID: \"d05e0184-6720-4dd3-a436-d9033446f7ac\") " pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.300433 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.300401    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:29.449072 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.449036    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9"]
Jul 14 15:22:29.452994 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.452971    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9"
Jul 14 15:22:29.459360 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.459332    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qqstz\""
Jul 14 15:22:29.460084 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.460056    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 14 15:22:29.461995 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.461969    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9"]
Jul 14 15:22:29.504759 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.504732    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-59799b7f6c-8zgxl"]
Jul 14 15:22:29.510792 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:29.510711    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05e0184_6720_4dd3_a436_d9033446f7ac.slice/crio-5b3b707bf60732cc7f10764b39759c5d59b00843fa5e5cc31149655b0bcdb30c WatchSource:0}: Error finding container 5b3b707bf60732cc7f10764b39759c5d59b00843fa5e5cc31149655b0bcdb30c: Status 404 returned error can't find the container with id 5b3b707bf60732cc7f10764b39759c5d59b00843fa5e5cc31149655b0bcdb30c
Jul 14 15:22:29.581733 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.581697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7k52l" event={"ID":"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2","Type":"ContainerStarted","Data":"4f362af49965ab7f5dc3bd0a73431c6a305d2be53c15e9ac9b2999e729c1498c"}
Jul 14 15:22:29.581903 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.581771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7k52l" event={"ID":"ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2","Type":"ContainerStarted","Data":"817a40c38271d8b477fb276134f117b0d74a4fb9954920afed674f9942cd4430"}
Jul 14 15:22:29.591838 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.591799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl" event={"ID":"d05e0184-6720-4dd3-a436-d9033446f7ac","Type":"ContainerStarted","Data":"5b3b707bf60732cc7f10764b39759c5d59b00843fa5e5cc31149655b0bcdb30c"}
Jul 14 15:22:29.594838 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.594813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr" event={"ID":"49e14489-568d-455c-97a2-241625d0155a","Type":"ContainerStarted","Data":"51ca737b9ae3b36156966801379b716c367defeebfb700a67067703224be4f66"}
Jul 14 15:22:29.594993 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.594866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr" event={"ID":"49e14489-568d-455c-97a2-241625d0155a","Type":"ContainerStarted","Data":"e582112a5ad4d678de4791cdf0f7e34ab30de7d92e6b0c2aaeb173f0c0921815"}
Jul 14 15:22:29.594993 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.594903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr" event={"ID":"49e14489-568d-455c-97a2-241625d0155a","Type":"ContainerStarted","Data":"71e86bd0f64841bab475051d0f7cd3156574864f3d04dbe1c292e498bc241fd3"}
Jul 14 15:22:29.596421 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.596401    2563 generic.go:356] "Generic (PLEG): container finished" podID="63ed7786-f31d-4538-a5d1-29b328cf5c23" containerID="ad2ae95fa2c5c062c6bf181d5e78f11bbe84f351be7f8e2ead55f0bf9324d7f3" exitCode=0
Jul 14 15:22:29.596533 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.596463    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63ed7786-f31d-4538-a5d1-29b328cf5c23","Type":"ContainerDied","Data":"ad2ae95fa2c5c062c6bf181d5e78f11bbe84f351be7f8e2ead55f0bf9324d7f3"}
Jul 14 15:22:29.596961 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.596933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlzmb\" (UniqueName: \"kubernetes.io/projected/aca900c5-8a1d-4571-b0cb-f0d8ce50f5df-kube-api-access-mlzmb\") pod \"monitoring-plugin-5f68df7f58-n8zd9\" (UID: \"aca900c5-8a1d-4571-b0cb-f0d8ce50f5df\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9"
Jul 14 15:22:29.597044 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.596995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aca900c5-8a1d-4571-b0cb-f0d8ce50f5df-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-n8zd9\" (UID: \"aca900c5-8a1d-4571-b0cb-f0d8ce50f5df\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9"
Jul 14 15:22:29.599616 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.599587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z" event={"ID":"95c86f2f-cfd8-4af0-ab06-4812eb0e8886","Type":"ContainerStarted","Data":"108fa20e05e64843d26fb1ab37cd10a605a63b29f4e32fcb0e48e90b952db2db"}
Jul 14 15:22:29.614308 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.614257    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7k52l" podStartSLOduration=4.679220237 podStartE2EDuration="5.614241175s" totalImagesPullingTime="847.702987ms" totalInitContainerRuntime="87.317951ms" isStatefulPod=true podCreationTimestamp="2026-07-14 15:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:29.612434359 +0000 UTC m=+73.953700292" watchObservedRunningTime="2026-07-14 15:22:29.614241175 +0000 UTC m=+73.955507081"
Jul 14 15:22:29.661731 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.661678    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-lc7kr" podStartSLOduration=3.875967221 podStartE2EDuration="5.661664107s" totalImagesPullingTime="1.785696886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:29.66090901 +0000 UTC m=+74.002174944" watchObservedRunningTime="2026-07-14 15:22:29.661664107 +0000 UTC m=+74.002929991"
Jul 14 15:22:29.680621 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.680574    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-w4r2z" podStartSLOduration=3.8022218580000002 podStartE2EDuration="5.680555921s" totalImagesPullingTime="1.878334063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:29.679759503 +0000 UTC m=+74.021025430" watchObservedRunningTime="2026-07-14 15:22:29.680555921 +0000 UTC m=+74.021821828"
Jul 14 15:22:29.698268 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.698232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlzmb\" (UniqueName: \"kubernetes.io/projected/aca900c5-8a1d-4571-b0cb-f0d8ce50f5df-kube-api-access-mlzmb\") pod \"monitoring-plugin-5f68df7f58-n8zd9\" (UID: \"aca900c5-8a1d-4571-b0cb-f0d8ce50f5df\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9"
Jul 14 15:22:29.698684 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.698655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aca900c5-8a1d-4571-b0cb-f0d8ce50f5df-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-n8zd9\" (UID: \"aca900c5-8a1d-4571-b0cb-f0d8ce50f5df\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9"
Jul 14 15:22:29.701450 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.701424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aca900c5-8a1d-4571-b0cb-f0d8ce50f5df-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-n8zd9\" (UID: \"aca900c5-8a1d-4571-b0cb-f0d8ce50f5df\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9"
Jul 14 15:22:29.707521 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.707489    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlzmb\" (UniqueName: \"kubernetes.io/projected/aca900c5-8a1d-4571-b0cb-f0d8ce50f5df-kube-api-access-mlzmb\") pod \"monitoring-plugin-5f68df7f58-n8zd9\" (UID: \"aca900c5-8a1d-4571-b0cb-f0d8ce50f5df\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9"
Jul 14 15:22:29.783107 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.783004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9"
Jul 14 15:22:29.917948 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:29.917732    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9"]
Jul 14 15:22:29.920902 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:29.920871    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca900c5_8a1d_4571_b0cb_f0d8ce50f5df.slice/crio-5c4c02859bf0c9f777f093d4a4b1462d6a4c0a618681ded73688dc256c463718 WatchSource:0}: Error finding container 5c4c02859bf0c9f777f093d4a4b1462d6a4c0a618681ded73688dc256c463718: Status 404 returned error can't find the container with id 5c4c02859bf0c9f777f093d4a4b1462d6a4c0a618681ded73688dc256c463718
Jul 14 15:22:30.609414 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.609384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9" event={"ID":"aca900c5-8a1d-4571-b0cb-f0d8ce50f5df","Type":"ContainerStarted","Data":"5c4c02859bf0c9f777f093d4a4b1462d6a4c0a618681ded73688dc256c463718"}
Jul 14 15:22:30.712905 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.712873    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f5b8cc457-ppbqm"]
Jul 14 15:22:30.716381 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.716293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.730380 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.730351    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f5b8cc457-ppbqm"]
Jul 14 15:22:30.809553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.809515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b294ba27-fa94-4e24-970b-33382e745581-console-oauth-config\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.809751 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.809588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-service-ca\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.809751 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.809624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b294ba27-fa94-4e24-970b-33382e745581-console-serving-cert\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.809751 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.809664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-oauth-serving-cert\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.809751 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.809709    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz58l\" (UniqueName: \"kubernetes.io/projected/b294ba27-fa94-4e24-970b-33382e745581-kube-api-access-lz58l\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.809954 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.809755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-console-config\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.809954 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.809819    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-trusted-ca-bundle\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.910856 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.910818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b294ba27-fa94-4e24-970b-33382e745581-console-serving-cert\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.910856 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.910867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-oauth-serving-cert\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.911052 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.910899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz58l\" (UniqueName: \"kubernetes.io/projected/b294ba27-fa94-4e24-970b-33382e745581-kube-api-access-lz58l\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.911052 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.910926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-console-config\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.911052 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.910966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-trusted-ca-bundle\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.911052 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.911023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b294ba27-fa94-4e24-970b-33382e745581-console-oauth-config\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.911201 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.911089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-service-ca\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.911822 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.911791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-console-config\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.912097 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.912042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-service-ca\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.912229 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.912133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-oauth-serving-cert\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.912229 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.912143    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-trusted-ca-bundle\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.913937 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.913916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b294ba27-fa94-4e24-970b-33382e745581-console-serving-cert\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.914329 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.914301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b294ba27-fa94-4e24-970b-33382e745581-console-oauth-config\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.921580 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.921552    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz58l\" (UniqueName: \"kubernetes.io/projected/b294ba27-fa94-4e24-970b-33382e745581-kube-api-access-lz58l\") pod \"console-7f5b8cc457-ppbqm\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") " pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:30.977687 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.977647    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 15:22:30.985433 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.985406    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:30.988558 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.988527    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 14 15:22:30.988558 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.988549    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 14 15:22:30.988815 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.988797    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 14 15:22:30.988958 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.988936    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 14 15:22:30.989084 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.989069    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 14 15:22:30.989206 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.989184    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 14 15:22:30.989258 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.989243    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2h7zc\""
Jul 14 15:22:30.989432 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.989415    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 14 15:22:30.991223 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.991201    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 14 15:22:30.995965 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.992990    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 14 15:22:30.995965 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.994603    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d0rf4jmad1rkf\""
Jul 14 15:22:30.995965 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.994838    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 14 15:22:30.995965 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.994935    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 14 15:22:30.995965 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:30.995169    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 14 15:22:31.004199 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.004077    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 14 15:22:31.008557 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.006740    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 15:22:31.008557 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.007357    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 14 15:22:31.034730 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.034691    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:31.112855 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.112816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113062 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.112876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113062 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.112901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113062 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.112952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113224 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b88eebc3-6119-4158-8e7a-d6d29f1c0745-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113224 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113224 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113224 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113224 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113404 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b88eebc3-6119-4158-8e7a-d6d29f1c0745-config-out\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113404 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113404 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113404 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113583 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113583 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113583 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg94m\" (UniqueName: \"kubernetes.io/projected/b88eebc3-6119-4158-8e7a-d6d29f1c0745-kube-api-access-hg94m\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113583 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113739 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-config\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113739 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.113739 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.113681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-web-config\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b88eebc3-6119-4158-8e7a-d6d29f1c0745-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b88eebc3-6119-4158-8e7a-d6d29f1c0745-config-out\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg94m\" (UniqueName: \"kubernetes.io/projected/b88eebc3-6119-4158-8e7a-d6d29f1c0745-kube-api-access-hg94m\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-config\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-web-config\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.216592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.217436 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.217436 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.217436 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.215793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.217436 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.216236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.218275 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.218246    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.219026 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.218998    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.219909 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.219519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.220995 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.220330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.220995 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.220674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b88eebc3-6119-4158-8e7a-d6d29f1c0745-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.220995 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.220702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-config\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.221199 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.221027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.221199 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.221093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.222101 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.222076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.223482 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.223445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b88eebc3-6119-4158-8e7a-d6d29f1c0745-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.224527 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.224076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b88eebc3-6119-4158-8e7a-d6d29f1c0745-config-out\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.224527 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.224237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-web-config\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.224527 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.224457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.224734 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.224639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.224734 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.224692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.229804 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.228855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.229804 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.229201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.230185 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.230145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b88eebc3-6119-4158-8e7a-d6d29f1c0745-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.231318 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.231296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg94m\" (UniqueName: \"kubernetes.io/projected/b88eebc3-6119-4158-8e7a-d6d29f1c0745-kube-api-access-hg94m\") pod \"prometheus-k8s-0\" (UID: \"b88eebc3-6119-4158-8e7a-d6d29f1c0745\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:31.306859 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:31.306589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:32.495370 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:32.495268    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 15:22:32.524760 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:32.523971    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f5b8cc457-ppbqm"]
Jul 14 15:22:32.531323 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:22:32.531290    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb294ba27_fa94_4e24_970b_33382e745581.slice/crio-b916401e72ee7fcc18c2e4306bd13eb4ab75a5e5b6fad82819f9ec839cc94138 WatchSource:0}: Error finding container b916401e72ee7fcc18c2e4306bd13eb4ab75a5e5b6fad82819f9ec839cc94138: Status 404 returned error can't find the container with id b916401e72ee7fcc18c2e4306bd13eb4ab75a5e5b6fad82819f9ec839cc94138
Jul 14 15:22:32.637263 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:32.636836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5b8cc457-ppbqm" event={"ID":"b294ba27-fa94-4e24-970b-33382e745581","Type":"ContainerStarted","Data":"b916401e72ee7fcc18c2e4306bd13eb4ab75a5e5b6fad82819f9ec839cc94138"}
Jul 14 15:22:32.647686 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:32.647115    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9" event={"ID":"aca900c5-8a1d-4571-b0cb-f0d8ce50f5df","Type":"ContainerStarted","Data":"e788e06c5bb93c51e7e74fa3ee5e2358012e1e50bb1bd90ad2dd6410067fda3b"}
Jul 14 15:22:32.647938 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:32.647896    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9"
Jul 14 15:22:32.654079 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:32.653960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7" event={"ID":"998a59ad-a466-47b3-a115-50318de410a3","Type":"ContainerStarted","Data":"d2478c206938d0c2567ef953c2fefa3ba41363d5b0cbf4d0011a6ebc8d5cc754"}
Jul 14 15:22:32.654079 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:32.654039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7" event={"ID":"998a59ad-a466-47b3-a115-50318de410a3","Type":"ContainerStarted","Data":"8f51ee8baa3741000642d35c2d74abfdb1e69dfd89bb1f757376865bb7f3f6e5"}
Jul 14 15:22:32.654079 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:32.654080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7" event={"ID":"998a59ad-a466-47b3-a115-50318de410a3","Type":"ContainerStarted","Data":"316cd5977466030c8883edb652cc13e116490214c63ad970bf2e04f61196c6cd"}
Jul 14 15:22:32.656806 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:32.656629    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9"
Jul 14 15:22:32.657822 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:32.657793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl" event={"ID":"d05e0184-6720-4dd3-a436-d9033446f7ac","Type":"ContainerStarted","Data":"495732dfc7322f5b438d69aec7d59fb0bdc5ffb3b838616a3c0e08c2a7ae0f66"}
Jul 14 15:22:32.662318 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:32.661520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b88eebc3-6119-4158-8e7a-d6d29f1c0745","Type":"ContainerStarted","Data":"4d45f608451e21303893c2387086606f08003f194dc99a1d9212a782047bda94"}
Jul 14 15:22:32.662318 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:32.661593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b88eebc3-6119-4158-8e7a-d6d29f1c0745","Type":"ContainerStarted","Data":"c788e1514dc39f6374c188da42ec042164b4534ca59c58247b85d0962dfa3609"}
Jul 14 15:22:32.668021 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:32.667595    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-n8zd9" podStartSLOduration=1.300886274 podStartE2EDuration="3.667493983s" totalImagesPullingTime="2.366607709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:32.665483751 +0000 UTC m=+77.006749677" watchObservedRunningTime="2026-07-14 15:22:32.667493983 +0000 UTC m=+77.008759891"
Jul 14 15:22:32.704414 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:32.704338    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl" podStartSLOduration=1.923436142 podStartE2EDuration="4.704313775s" totalImagesPullingTime="2.780877633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:32.70217245 +0000 UTC m=+77.043438357" watchObservedRunningTime="2026-07-14 15:22:32.704313775 +0000 UTC m=+77.045579684"
Jul 14 15:22:33.667986 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:33.667949    2563 generic.go:356] "Generic (PLEG): container finished" podID="b88eebc3-6119-4158-8e7a-d6d29f1c0745" containerID="4d45f608451e21303893c2387086606f08003f194dc99a1d9212a782047bda94" exitCode=0
Jul 14 15:22:33.668389 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:33.668060    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b88eebc3-6119-4158-8e7a-d6d29f1c0745","Type":"ContainerDied","Data":"4d45f608451e21303893c2387086606f08003f194dc99a1d9212a782047bda94"}
Jul 14 15:22:33.676318 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:33.676289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63ed7786-f31d-4538-a5d1-29b328cf5c23","Type":"ContainerStarted","Data":"a5ae5b676cb902531db9d4ce2d727734efda03a28b510e7f55b073f0e44b1fb4"}
Jul 14 15:22:33.676458 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:33.676397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63ed7786-f31d-4538-a5d1-29b328cf5c23","Type":"ContainerStarted","Data":"6d3fb99c95945b4ccd0bb272fac1476951a569bb3a0c870a84c3d424494471bf"}
Jul 14 15:22:33.676458 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:33.676439    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63ed7786-f31d-4538-a5d1-29b328cf5c23","Type":"ContainerStarted","Data":"6f373b187e8171e313a55fa39097bb2a34e3f99b311a86bc0382f6f9721fb895"}
Jul 14 15:22:33.676577 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:33.676475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63ed7786-f31d-4538-a5d1-29b328cf5c23","Type":"ContainerStarted","Data":"c4a51fa6f691df67563567bf298f4d28fcdd92e93d253e3580dd36e4fba0811b"}
Jul 14 15:22:33.678048 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:33.678024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5b8cc457-ppbqm" event={"ID":"b294ba27-fa94-4e24-970b-33382e745581","Type":"ContainerStarted","Data":"2d7038ff0807cc46dca71d033f42f270ed08723b0a3c8a0f2f8618449c8a973e"}
Jul 14 15:22:34.687468 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:34.687427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63ed7786-f31d-4538-a5d1-29b328cf5c23","Type":"ContainerStarted","Data":"a93ca2514c6916262bb9abd152519be9772a9c3ccb70dcd2ada90f12727f52d4"}
Jul 14 15:22:34.687927 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:34.687528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63ed7786-f31d-4538-a5d1-29b328cf5c23","Type":"ContainerStarted","Data":"240e62c166072a5d2dd8edc857c13bb858364dcbb151512d89ea8f257e6c4410"}
Jul 14 15:22:34.691328 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:34.691291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7" event={"ID":"998a59ad-a466-47b3-a115-50318de410a3","Type":"ContainerStarted","Data":"effdf095d0d427c2f2fec234b61384bbeb1d4212f827bb506e3b23f1c9b87b1c"}
Jul 14 15:22:34.691482 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:34.691367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7" event={"ID":"998a59ad-a466-47b3-a115-50318de410a3","Type":"ContainerStarted","Data":"54bd1794267f2092651b7d7b5892e0e6ed3705c32f1c843692a59bbcf0359080"}
Jul 14 15:22:34.691482 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:34.691406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7" event={"ID":"998a59ad-a466-47b3-a115-50318de410a3","Type":"ContainerStarted","Data":"e4002857965bebf25762a7aaaa60f0fac5961668ff36bcda00602264d68f1f43"}
Jul 14 15:22:34.719566 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:34.719518    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f5b8cc457-ppbqm" podStartSLOduration=4.719484975 podStartE2EDuration="4.719484975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:33.71392907 +0000 UTC m=+78.055194979" watchObservedRunningTime="2026-07-14 15:22:34.719484975 +0000 UTC m=+79.060750861"
Jul 14 15:22:34.719900 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:34.719882    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.3346019590000004 podStartE2EDuration="9.719873725s" totalImagesPullingTime="5.366636874s" totalInitContainerRuntime="18.634892ms" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:34.717613149 +0000 UTC m=+79.058879073" watchObservedRunningTime="2026-07-14 15:22:34.719873725 +0000 UTC m=+79.061139632"
Jul 14 15:22:34.744443 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:34.744384    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7" podStartSLOduration=2.513428745 podStartE2EDuration="7.744368943s" totalImagesPullingTime="5.230940198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:34.743557855 +0000 UTC m=+79.084823795" watchObservedRunningTime="2026-07-14 15:22:34.744368943 +0000 UTC m=+79.085634850"
Jul 14 15:22:35.698246 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:35.698198    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:37.711026 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:37.710991    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b88eebc3-6119-4158-8e7a-d6d29f1c0745","Type":"ContainerStarted","Data":"eb7ae64c0e43371f247f7790edc32fd9069ad396c0a561984f30494312b28145"}
Jul 14 15:22:37.711458 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:37.711093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b88eebc3-6119-4158-8e7a-d6d29f1c0745","Type":"ContainerStarted","Data":"1db03d440aef324e4307f4e6ba33fc86223bd66f8f5aa6a06d7fc90fb8f336b3"}
Jul 14 15:22:37.711458 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:37.711121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b88eebc3-6119-4158-8e7a-d6d29f1c0745","Type":"ContainerStarted","Data":"1b12fdf4827b4db8fa322309c727a0452864d8b6c779d9ef56c21e323c867df9"}
Jul 14 15:22:37.711458 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:37.711146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b88eebc3-6119-4158-8e7a-d6d29f1c0745","Type":"ContainerStarted","Data":"f357b1d0acba061bb95d492ced77f3b9b8cc0815595d1ba72379026273dd586e"}
Jul 14 15:22:37.711458 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:37.711172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b88eebc3-6119-4158-8e7a-d6d29f1c0745","Type":"ContainerStarted","Data":"611de871d878b66a3b0c5f7433a7d3ffe8e0231cf4e72e5ab60f1f727e899c30"}
Jul 14 15:22:37.711458 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:37.711198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b88eebc3-6119-4158-8e7a-d6d29f1c0745","Type":"ContainerStarted","Data":"ee15c15c7c52911ecaa0c412d3137e4c0b60ab411c01dbbd5b71569b5bec8653"}
Jul 14 15:22:37.743530 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:37.743446    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.3863406 podStartE2EDuration="7.743425084s" totalImagesPullingTime="3.313067966s" totalInitContainerRuntime="44.016518ms" isStatefulPod=false podCreationTimestamp="2026-07-14 15:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:22:37.74103844 +0000 UTC m=+82.082304386" watchObservedRunningTime="2026-07-14 15:22:37.743425084 +0000 UTC m=+82.084690991"
Jul 14 15:22:41.035478 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:41.035438    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:41.035973 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:41.035523    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:41.040532 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:41.040496    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:41.307385 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:41.307282    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:22:41.710860 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:41.710835    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-69748d48dc-wspq7"
Jul 14 15:22:41.734638 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:41.734603    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:22:41.786297 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:41.786263    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-746764f8c4-4p5q2"]
Jul 14 15:22:49.300990 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:49.300952    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:49.301461 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:49.301012    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:22:53.668294 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:53.668231    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58949658f4-rm47p" podUID="213912a0-c209-4463-b314-654cb2d17075" containerName="console" containerID="cri-o://91e89e77410f7671800f4284124878359bddda5cdb44f500eb3fa9a5fb0864d3" gracePeriod=15
Jul 14 15:22:53.923224 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:53.923161    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:53.930571 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:53.930549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/213912a0-c209-4463-b314-654cb2d17075-console-serving-cert\") pod \"213912a0-c209-4463-b314-654cb2d17075\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") "
Jul 14 15:22:53.930673 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:53.930582    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-console-config\") pod \"213912a0-c209-4463-b314-654cb2d17075\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") "
Jul 14 15:22:53.930673 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:53.930612    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-service-ca\") pod \"213912a0-c209-4463-b314-654cb2d17075\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") "
Jul 14 15:22:53.930673 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:53.930668    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-oauth-serving-cert\") pod \"213912a0-c209-4463-b314-654cb2d17075\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") "
Jul 14 15:22:53.930818 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:53.930741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78ms7\" (UniqueName: \"kubernetes.io/projected/213912a0-c209-4463-b314-654cb2d17075-kube-api-access-78ms7\") pod \"213912a0-c209-4463-b314-654cb2d17075\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") "
Jul 14 15:22:53.930818 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:53.930774    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/213912a0-c209-4463-b314-654cb2d17075-console-oauth-config\") pod \"213912a0-c209-4463-b314-654cb2d17075\" (UID: \"213912a0-c209-4463-b314-654cb2d17075\") "
Jul 14 15:22:53.931041 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:53.931015    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-console-config" (OuterVolumeSpecName: "console-config") pod "213912a0-c209-4463-b314-654cb2d17075" (UID: "213912a0-c209-4463-b314-654cb2d17075"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:22:53.931041 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:53.931026    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-service-ca" (OuterVolumeSpecName: "service-ca") pod "213912a0-c209-4463-b314-654cb2d17075" (UID: "213912a0-c209-4463-b314-654cb2d17075"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:22:53.931174 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:53.931115    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "213912a0-c209-4463-b314-654cb2d17075" (UID: "213912a0-c209-4463-b314-654cb2d17075"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:22:53.933069 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:53.933041    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213912a0-c209-4463-b314-654cb2d17075-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "213912a0-c209-4463-b314-654cb2d17075" (UID: "213912a0-c209-4463-b314-654cb2d17075"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:22:53.933179 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:53.933157    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213912a0-c209-4463-b314-654cb2d17075-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "213912a0-c209-4463-b314-654cb2d17075" (UID: "213912a0-c209-4463-b314-654cb2d17075"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:22:53.933237 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:53.933187    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213912a0-c209-4463-b314-654cb2d17075-kube-api-access-78ms7" (OuterVolumeSpecName: "kube-api-access-78ms7") pod "213912a0-c209-4463-b314-654cb2d17075" (UID: "213912a0-c209-4463-b314-654cb2d17075"). InnerVolumeSpecName "kube-api-access-78ms7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:22:54.031590 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.031547    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/213912a0-c209-4463-b314-654cb2d17075-console-serving-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:22:54.031590 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.031579    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-console-config\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:22:54.031590 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.031589    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:22:54.031590 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.031597    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/213912a0-c209-4463-b314-654cb2d17075-oauth-serving-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:22:54.031856 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.031607    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78ms7\" (UniqueName: \"kubernetes.io/projected/213912a0-c209-4463-b314-654cb2d17075-kube-api-access-78ms7\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:22:54.031856 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.031616    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/213912a0-c209-4463-b314-654cb2d17075-console-oauth-config\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:22:54.786704 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.786661    2563 generic.go:356] "Generic (PLEG): container finished" podID="213912a0-c209-4463-b314-654cb2d17075" containerID="91e89e77410f7671800f4284124878359bddda5cdb44f500eb3fa9a5fb0864d3" exitCode=0
Jul 14 15:22:54.787105 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.786712    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58949658f4-rm47p"
Jul 14 15:22:54.787105 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.786743    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58949658f4-rm47p" event={"ID":"213912a0-c209-4463-b314-654cb2d17075","Type":"ContainerDied","Data":"91e89e77410f7671800f4284124878359bddda5cdb44f500eb3fa9a5fb0864d3"}
Jul 14 15:22:54.787105 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.786793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58949658f4-rm47p" event={"ID":"213912a0-c209-4463-b314-654cb2d17075","Type":"ContainerDied","Data":"8816a641f1354ac802b494fb1a4242bcbb98f9859d8cc741a2696530bf95efd2"}
Jul 14 15:22:54.787105 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.786825    2563 scope.go:117] "RemoveContainer" containerID="91e89e77410f7671800f4284124878359bddda5cdb44f500eb3fa9a5fb0864d3"
Jul 14 15:22:54.795621 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.795606    2563 scope.go:117] "RemoveContainer" containerID="91e89e77410f7671800f4284124878359bddda5cdb44f500eb3fa9a5fb0864d3"
Jul 14 15:22:54.795949 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:22:54.795916    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e89e77410f7671800f4284124878359bddda5cdb44f500eb3fa9a5fb0864d3\": container with ID starting with 91e89e77410f7671800f4284124878359bddda5cdb44f500eb3fa9a5fb0864d3 not found: ID does not exist" containerID="91e89e77410f7671800f4284124878359bddda5cdb44f500eb3fa9a5fb0864d3"
Jul 14 15:22:54.795994 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.795959    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e89e77410f7671800f4284124878359bddda5cdb44f500eb3fa9a5fb0864d3"} err="failed to get container status \"91e89e77410f7671800f4284124878359bddda5cdb44f500eb3fa9a5fb0864d3\": rpc error: code = NotFound desc = could not find container \"91e89e77410f7671800f4284124878359bddda5cdb44f500eb3fa9a5fb0864d3\": container with ID starting with 91e89e77410f7671800f4284124878359bddda5cdb44f500eb3fa9a5fb0864d3 not found: ID does not exist"
Jul 14 15:22:54.807150 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.807124    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58949658f4-rm47p"]
Jul 14 15:22:54.808672 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:54.808647    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58949658f4-rm47p"]
Jul 14 15:22:56.219152 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:56.219116    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="213912a0-c209-4463-b314-654cb2d17075" path="/var/lib/kubelet/pods/213912a0-c209-4463-b314-654cb2d17075/volumes"
Jul 14 15:22:58.577457 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:22:58.577424    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w82g9"
Jul 14 15:23:06.807662 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:06.807599    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-746764f8c4-4p5q2" podUID="7f4a7344-19e2-4ae6-ac29-cffbed75b5b2" containerName="console" containerID="cri-o://a242ee73a1e4224396c4ee9a8963d5fc218da0d32f78bce024d6080339dc6e8a" gracePeriod=15
Jul 14 15:23:07.113464 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.113434    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:23:07.128889 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.128859    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-oauth-config\") pod \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") "
Jul 14 15:23:07.129072 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.128899    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-serving-cert\") pod \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") "
Jul 14 15:23:07.129072 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.128960    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-service-ca\") pod \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") "
Jul 14 15:23:07.129072 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.128996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-config\") pod \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") "
Jul 14 15:23:07.129072 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.129027    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-trusted-ca-bundle\") pod \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") "
Jul 14 15:23:07.129072 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.129052    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g55c\" (UniqueName: \"kubernetes.io/projected/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-kube-api-access-5g55c\") pod \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") "
Jul 14 15:23:07.129305 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.129110    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-oauth-serving-cert\") pod \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\" (UID: \"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2\") "
Jul 14 15:23:07.129403 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.129368    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-service-ca" (OuterVolumeSpecName: "service-ca") pod "7f4a7344-19e2-4ae6-ac29-cffbed75b5b2" (UID: "7f4a7344-19e2-4ae6-ac29-cffbed75b5b2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:23:07.129560 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.129531    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-config" (OuterVolumeSpecName: "console-config") pod "7f4a7344-19e2-4ae6-ac29-cffbed75b5b2" (UID: "7f4a7344-19e2-4ae6-ac29-cffbed75b5b2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:23:07.129860 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.129634    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7f4a7344-19e2-4ae6-ac29-cffbed75b5b2" (UID: "7f4a7344-19e2-4ae6-ac29-cffbed75b5b2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:23:07.129860 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.129831    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7f4a7344-19e2-4ae6-ac29-cffbed75b5b2" (UID: "7f4a7344-19e2-4ae6-ac29-cffbed75b5b2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:23:07.131431 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.131403    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7f4a7344-19e2-4ae6-ac29-cffbed75b5b2" (UID: "7f4a7344-19e2-4ae6-ac29-cffbed75b5b2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:23:07.131703 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.131498    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7f4a7344-19e2-4ae6-ac29-cffbed75b5b2" (UID: "7f4a7344-19e2-4ae6-ac29-cffbed75b5b2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:23:07.131703 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.131556    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-kube-api-access-5g55c" (OuterVolumeSpecName: "kube-api-access-5g55c") pod "7f4a7344-19e2-4ae6-ac29-cffbed75b5b2" (UID: "7f4a7344-19e2-4ae6-ac29-cffbed75b5b2"). InnerVolumeSpecName "kube-api-access-5g55c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:23:07.229967 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.229938    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:23:07.229967 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.229968    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-config\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:23:07.230163 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.229977    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-trusted-ca-bundle\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:23:07.230163 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.229986    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g55c\" (UniqueName: \"kubernetes.io/projected/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-kube-api-access-5g55c\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:23:07.230163 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.229999    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-oauth-serving-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:23:07.230163 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.230010    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-oauth-config\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:23:07.230163 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.230022    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2-console-serving-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:23:07.845610 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.845577    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f4a7344-19e2-4ae6-ac29-cffbed75b5b2" containerID="a242ee73a1e4224396c4ee9a8963d5fc218da0d32f78bce024d6080339dc6e8a" exitCode=0
Jul 14 15:23:07.846062 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.845628    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746764f8c4-4p5q2"
Jul 14 15:23:07.846062 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.845647    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746764f8c4-4p5q2" event={"ID":"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2","Type":"ContainerDied","Data":"a242ee73a1e4224396c4ee9a8963d5fc218da0d32f78bce024d6080339dc6e8a"}
Jul 14 15:23:07.846062 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.845708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746764f8c4-4p5q2" event={"ID":"7f4a7344-19e2-4ae6-ac29-cffbed75b5b2","Type":"ContainerDied","Data":"214289cb606cb3b6bfbaf3fe91bcae750e3f7e1a1e251379008943947e60bf13"}
Jul 14 15:23:07.846062 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.845731    2563 scope.go:117] "RemoveContainer" containerID="a242ee73a1e4224396c4ee9a8963d5fc218da0d32f78bce024d6080339dc6e8a"
Jul 14 15:23:07.855083 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.855066    2563 scope.go:117] "RemoveContainer" containerID="a242ee73a1e4224396c4ee9a8963d5fc218da0d32f78bce024d6080339dc6e8a"
Jul 14 15:23:07.855392 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:23:07.855373    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a242ee73a1e4224396c4ee9a8963d5fc218da0d32f78bce024d6080339dc6e8a\": container with ID starting with a242ee73a1e4224396c4ee9a8963d5fc218da0d32f78bce024d6080339dc6e8a not found: ID does not exist" containerID="a242ee73a1e4224396c4ee9a8963d5fc218da0d32f78bce024d6080339dc6e8a"
Jul 14 15:23:07.855471 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.855401    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a242ee73a1e4224396c4ee9a8963d5fc218da0d32f78bce024d6080339dc6e8a"} err="failed to get container status \"a242ee73a1e4224396c4ee9a8963d5fc218da0d32f78bce024d6080339dc6e8a\": rpc error: code = NotFound desc = could not find container \"a242ee73a1e4224396c4ee9a8963d5fc218da0d32f78bce024d6080339dc6e8a\": container with ID starting with a242ee73a1e4224396c4ee9a8963d5fc218da0d32f78bce024d6080339dc6e8a not found: ID does not exist"
Jul 14 15:23:07.869447 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.869415    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-746764f8c4-4p5q2"]
Jul 14 15:23:07.871718 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:07.871692    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-746764f8c4-4p5q2"]
Jul 14 15:23:08.216407 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:08.216377    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f4a7344-19e2-4ae6-ac29-cffbed75b5b2" path="/var/lib/kubelet/pods/7f4a7344-19e2-4ae6-ac29-cffbed75b5b2/volumes"
Jul 14 15:23:09.307056 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:09.307027    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:23:09.312236 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:09.312208    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-59799b7f6c-8zgxl"
Jul 14 15:23:31.307084 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:31.307044    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:23:31.327835 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:31.327808    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:23:31.968629 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:31.968603    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 15:23:49.023056 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.023024    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7599546c9b-5sjmc"]
Jul 14 15:23:49.023695 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.023558    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f4a7344-19e2-4ae6-ac29-cffbed75b5b2" containerName="console"
Jul 14 15:23:49.023695 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.023579    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f4a7344-19e2-4ae6-ac29-cffbed75b5b2" containerName="console"
Jul 14 15:23:49.023695 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.023598    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="213912a0-c209-4463-b314-654cb2d17075" containerName="console"
Jul 14 15:23:49.023695 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.023607    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="213912a0-c209-4463-b314-654cb2d17075" containerName="console"
Jul 14 15:23:49.023695 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.023688    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f4a7344-19e2-4ae6-ac29-cffbed75b5b2" containerName="console"
Jul 14 15:23:49.023963 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.023703    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="213912a0-c209-4463-b314-654cb2d17075" containerName="console"
Jul 14 15:23:49.026224 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.026205    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.029574 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.029549    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 14 15:23:49.029749 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.029548    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 14 15:23:49.029749 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.029625    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6kgvn\""
Jul 14 15:23:49.029913 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.029660    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 14 15:23:49.029913 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.029685    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 14 15:23:49.030011 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.029702    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 14 15:23:49.035302 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.035278    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 14 15:23:49.037692 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.037672    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7599546c9b-5sjmc"]
Jul 14 15:23:49.101459 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.101431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/817a51df-0e8e-4a0e-af15-981b22389a84-metrics-client-ca\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.101671 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.101495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/817a51df-0e8e-4a0e-af15-981b22389a84-federate-client-tls\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.101671 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.101576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817a51df-0e8e-4a0e-af15-981b22389a84-serving-certs-ca-bundle\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.101671 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.101605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817a51df-0e8e-4a0e-af15-981b22389a84-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.101789 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.101680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/817a51df-0e8e-4a0e-af15-981b22389a84-secret-telemeter-client\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.101789 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.101715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/817a51df-0e8e-4a0e-af15-981b22389a84-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.101789 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.101740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgfcp\" (UniqueName: \"kubernetes.io/projected/817a51df-0e8e-4a0e-af15-981b22389a84-kube-api-access-mgfcp\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.101789 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.101760    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/817a51df-0e8e-4a0e-af15-981b22389a84-telemeter-client-tls\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.203052 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.203009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/817a51df-0e8e-4a0e-af15-981b22389a84-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.203251 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.203061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgfcp\" (UniqueName: \"kubernetes.io/projected/817a51df-0e8e-4a0e-af15-981b22389a84-kube-api-access-mgfcp\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.203251 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.203090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/817a51df-0e8e-4a0e-af15-981b22389a84-telemeter-client-tls\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.203251 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.203123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/817a51df-0e8e-4a0e-af15-981b22389a84-metrics-client-ca\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.203251 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.203174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/817a51df-0e8e-4a0e-af15-981b22389a84-federate-client-tls\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.203251 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.203210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817a51df-0e8e-4a0e-af15-981b22389a84-serving-certs-ca-bundle\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.203251 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.203230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817a51df-0e8e-4a0e-af15-981b22389a84-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.203586 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.203281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/817a51df-0e8e-4a0e-af15-981b22389a84-secret-telemeter-client\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.204022 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.203995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/817a51df-0e8e-4a0e-af15-981b22389a84-metrics-client-ca\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.204367 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.204172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817a51df-0e8e-4a0e-af15-981b22389a84-serving-certs-ca-bundle\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.204367 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.204357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817a51df-0e8e-4a0e-af15-981b22389a84-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.205836 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.205805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/817a51df-0e8e-4a0e-af15-981b22389a84-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.205949 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.205933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/817a51df-0e8e-4a0e-af15-981b22389a84-federate-client-tls\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.205990 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.205961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/817a51df-0e8e-4a0e-af15-981b22389a84-telemeter-client-tls\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.206024 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.206008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/817a51df-0e8e-4a0e-af15-981b22389a84-secret-telemeter-client\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.212566 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.212534    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgfcp\" (UniqueName: \"kubernetes.io/projected/817a51df-0e8e-4a0e-af15-981b22389a84-kube-api-access-mgfcp\") pod \"telemeter-client-7599546c9b-5sjmc\" (UID: \"817a51df-0e8e-4a0e-af15-981b22389a84\") " pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.338131 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.338030    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc"
Jul 14 15:23:49.477198 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:49.477173    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7599546c9b-5sjmc"]
Jul 14 15:23:49.479673 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:23:49.479642    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817a51df_0e8e_4a0e_af15_981b22389a84.slice/crio-ec7c497c5e7fbdb512e9bc0fd3540d66d93d7d20a09eb6e5f297fb8d3540fbdd WatchSource:0}: Error finding container ec7c497c5e7fbdb512e9bc0fd3540d66d93d7d20a09eb6e5f297fb8d3540fbdd: Status 404 returned error can't find the container with id ec7c497c5e7fbdb512e9bc0fd3540d66d93d7d20a09eb6e5f297fb8d3540fbdd
Jul 14 15:23:50.025654 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:50.025623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc" event={"ID":"817a51df-0e8e-4a0e-af15-981b22389a84","Type":"ContainerStarted","Data":"ec7c497c5e7fbdb512e9bc0fd3540d66d93d7d20a09eb6e5f297fb8d3540fbdd"}
Jul 14 15:23:52.036622 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:52.036593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc" event={"ID":"817a51df-0e8e-4a0e-af15-981b22389a84","Type":"ContainerStarted","Data":"3fe1ea4136e18a134c4ab869c02c86d10f7e2d0e8ae30130710152ad6ea0c629"}
Jul 14 15:23:52.036932 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:52.036659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc" event={"ID":"817a51df-0e8e-4a0e-af15-981b22389a84","Type":"ContainerStarted","Data":"212b51706470369c62861ab0dd9d80cfea59a57b2ed63a99fca3e7b8b02d46e7"}
Jul 14 15:23:53.042121 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:53.042086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc" event={"ID":"817a51df-0e8e-4a0e-af15-981b22389a84","Type":"ContainerStarted","Data":"1f94d4602147a1a90598d83ff91e9c5ccae211668572d3a6d10211f9d0493c81"}
Jul 14 15:23:53.066963 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:53.066908    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7599546c9b-5sjmc" podStartSLOduration=1.654201789 podStartE2EDuration="4.066892449s" totalImagesPullingTime="2.41269066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:23:53.064984712 +0000 UTC m=+157.406250674" watchObservedRunningTime="2026-07-14 15:23:53.066892449 +0000 UTC m=+157.408158357"
Jul 14 15:23:54.031111 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.031079    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-695f5bb85d-hbmjs"]
Jul 14 15:23:54.033788 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.033769    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.046053 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.046023    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-695f5bb85d-hbmjs"]
Jul 14 15:23:54.147364 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.147326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-serving-cert\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.147364 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.147367    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-service-ca\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.147727 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.147418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-config\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.147727 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.147477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-oauth-config\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.147727 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.147562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-trusted-ca-bundle\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.147727 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.147673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-oauth-serving-cert\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.147932 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.147766    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj4vq\" (UniqueName: \"kubernetes.io/projected/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-kube-api-access-xj4vq\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.249137 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.249089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-oauth-config\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.249398 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.249172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-trusted-ca-bundle\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.249398 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.249225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-oauth-serving-cert\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.249398 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.249250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj4vq\" (UniqueName: \"kubernetes.io/projected/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-kube-api-access-xj4vq\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.249398 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.249333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-serving-cert\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.249398 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.249357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-service-ca\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.249398 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.249387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-config\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.250033 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.250008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-oauth-serving-cert\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.250146 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.250124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-config\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.250194 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.250163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-service-ca\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.250309 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.250292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-trusted-ca-bundle\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.252638 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.252614    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-oauth-config\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.252743 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.252617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-serving-cert\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.258730 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.258703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj4vq\" (UniqueName: \"kubernetes.io/projected/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-kube-api-access-xj4vq\") pod \"console-695f5bb85d-hbmjs\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") " pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.346430 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.346341    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:23:54.498601 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:54.498577    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-695f5bb85d-hbmjs"]
Jul 14 15:23:55.054082 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:55.054045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695f5bb85d-hbmjs" event={"ID":"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6","Type":"ContainerStarted","Data":"e873caf855e29bc2f48916793c3eab296646f36388d5e1f1ff4d2368779253bb"}
Jul 14 15:23:55.054459 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:55.054096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695f5bb85d-hbmjs" event={"ID":"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6","Type":"ContainerStarted","Data":"a8013b09cdf57f7dcc10fb52fa726fd8e5c78d20eca474ba521c6bbbf9b964ca"}
Jul 14 15:23:55.075176 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:23:55.075113    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-695f5bb85d-hbmjs" podStartSLOduration=1.075097933 podStartE2EDuration="1.075097933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:23:55.072966567 +0000 UTC m=+159.414232489" watchObservedRunningTime="2026-07-14 15:23:55.075097933 +0000 UTC m=+159.416363914"
Jul 14 15:24:04.347306 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:04.347252    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:24:04.347306 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:04.347322    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:24:04.352182 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:04.352156    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:24:05.098458 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.098430    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:24:05.147518 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.147472    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f5b8cc457-ppbqm"]
Jul 14 15:24:05.218551 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.218518    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67f7c8ff86-dbtwv"]
Jul 14 15:24:05.224980 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.224950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.231211 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.231185    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f7c8ff86-dbtwv"]
Jul 14 15:24:05.359000 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.358897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-config\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.359000 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.358937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-oauth-serving-cert\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.359000 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.358971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cxlk\" (UniqueName: \"kubernetes.io/projected/88a4571e-bc89-4552-99d9-ac3747bbbec8-kube-api-access-9cxlk\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.359494 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.359074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-trusted-ca-bundle\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.359494 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.359115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-serving-cert\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.359494 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.359193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-oauth-config\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.359494 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.359219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-service-ca\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.459852 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.459816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cxlk\" (UniqueName: \"kubernetes.io/projected/88a4571e-bc89-4552-99d9-ac3747bbbec8-kube-api-access-9cxlk\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.460067 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.459878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-trusted-ca-bundle\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.460067 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.459896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-serving-cert\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.460067 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.459925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-oauth-config\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.460067 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.459945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-service-ca\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.460067 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.459983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-config\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.460067 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.459997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-oauth-serving-cert\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.460717 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.460692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-service-ca\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.460997 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.460781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-config\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.460997 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.460853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-oauth-serving-cert\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.461118 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.461022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-trusted-ca-bundle\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.462644 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.462620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-oauth-config\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.462730 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.462702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-serving-cert\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.468574 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.468541    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cxlk\" (UniqueName: \"kubernetes.io/projected/88a4571e-bc89-4552-99d9-ac3747bbbec8-kube-api-access-9cxlk\") pod \"console-67f7c8ff86-dbtwv\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") " pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.535949 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.535907    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:05.672856 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:05.672832    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f7c8ff86-dbtwv"]
Jul 14 15:24:05.674744 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:24:05.674718    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a4571e_bc89_4552_99d9_ac3747bbbec8.slice/crio-24306f966672604c8e2452562b4ac5b37d720560a69941d0ee9b054b712b5f89 WatchSource:0}: Error finding container 24306f966672604c8e2452562b4ac5b37d720560a69941d0ee9b054b712b5f89: Status 404 returned error can't find the container with id 24306f966672604c8e2452562b4ac5b37d720560a69941d0ee9b054b712b5f89
Jul 14 15:24:06.100678 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:06.100635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7c8ff86-dbtwv" event={"ID":"88a4571e-bc89-4552-99d9-ac3747bbbec8","Type":"ContainerStarted","Data":"558ef529829f6719ce73b97c365a08fb4e33709857c8949c7dd42b0104dfd3dc"}
Jul 14 15:24:06.100877 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:06.100698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7c8ff86-dbtwv" event={"ID":"88a4571e-bc89-4552-99d9-ac3747bbbec8","Type":"ContainerStarted","Data":"24306f966672604c8e2452562b4ac5b37d720560a69941d0ee9b054b712b5f89"}
Jul 14 15:24:06.121673 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:06.121610    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67f7c8ff86-dbtwv" podStartSLOduration=1.121591058 podStartE2EDuration="1.121591058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:24:06.120214265 +0000 UTC m=+170.461480173" watchObservedRunningTime="2026-07-14 15:24:06.121591058 +0000 UTC m=+170.462856964"
Jul 14 15:24:15.536478 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:15.536382    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:15.536478 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:15.536486    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:15.541241 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:15.541216    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:16.148568 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:16.148545    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:24:16.229401 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:16.229372    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-695f5bb85d-hbmjs"]
Jul 14 15:24:30.175420 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.175357    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f5b8cc457-ppbqm" podUID="b294ba27-fa94-4e24-970b-33382e745581" containerName="console" containerID="cri-o://2d7038ff0807cc46dca71d033f42f270ed08723b0a3c8a0f2f8618449c8a973e" gracePeriod=15
Jul 14 15:24:30.416576 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.416554    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:24:30.483686 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.483617    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-oauth-serving-cert\") pod \"b294ba27-fa94-4e24-970b-33382e745581\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") "
Jul 14 15:24:30.483816 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.483688    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-console-config\") pod \"b294ba27-fa94-4e24-970b-33382e745581\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") "
Jul 14 15:24:30.483816 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.483728    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-trusted-ca-bundle\") pod \"b294ba27-fa94-4e24-970b-33382e745581\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") "
Jul 14 15:24:30.483816 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.483758    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b294ba27-fa94-4e24-970b-33382e745581-console-oauth-config\") pod \"b294ba27-fa94-4e24-970b-33382e745581\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") "
Jul 14 15:24:30.483816 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.483782    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-service-ca\") pod \"b294ba27-fa94-4e24-970b-33382e745581\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") "
Jul 14 15:24:30.484060 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.483847    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b294ba27-fa94-4e24-970b-33382e745581-console-serving-cert\") pod \"b294ba27-fa94-4e24-970b-33382e745581\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") "
Jul 14 15:24:30.484060 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.483892    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz58l\" (UniqueName: \"kubernetes.io/projected/b294ba27-fa94-4e24-970b-33382e745581-kube-api-access-lz58l\") pod \"b294ba27-fa94-4e24-970b-33382e745581\" (UID: \"b294ba27-fa94-4e24-970b-33382e745581\") "
Jul 14 15:24:30.484161 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.484027    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b294ba27-fa94-4e24-970b-33382e745581" (UID: "b294ba27-fa94-4e24-970b-33382e745581"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:24:30.484217 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.484158    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-console-config" (OuterVolumeSpecName: "console-config") pod "b294ba27-fa94-4e24-970b-33382e745581" (UID: "b294ba27-fa94-4e24-970b-33382e745581"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:24:30.484217 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.484173    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b294ba27-fa94-4e24-970b-33382e745581" (UID: "b294ba27-fa94-4e24-970b-33382e745581"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:24:30.484217 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.484183    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-service-ca" (OuterVolumeSpecName: "service-ca") pod "b294ba27-fa94-4e24-970b-33382e745581" (UID: "b294ba27-fa94-4e24-970b-33382e745581"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:24:30.486041 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.486014    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b294ba27-fa94-4e24-970b-33382e745581-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b294ba27-fa94-4e24-970b-33382e745581" (UID: "b294ba27-fa94-4e24-970b-33382e745581"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:24:30.486144 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.486062    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b294ba27-fa94-4e24-970b-33382e745581-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b294ba27-fa94-4e24-970b-33382e745581" (UID: "b294ba27-fa94-4e24-970b-33382e745581"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:24:30.486144 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.486089    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b294ba27-fa94-4e24-970b-33382e745581-kube-api-access-lz58l" (OuterVolumeSpecName: "kube-api-access-lz58l") pod "b294ba27-fa94-4e24-970b-33382e745581" (UID: "b294ba27-fa94-4e24-970b-33382e745581"). InnerVolumeSpecName "kube-api-access-lz58l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:24:30.584606 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.584571    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-console-config\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:24:30.584606 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.584601    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-trusted-ca-bundle\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:24:30.584606 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.584613    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b294ba27-fa94-4e24-970b-33382e745581-console-oauth-config\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:24:30.584837 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.584621    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:24:30.584837 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.584630    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b294ba27-fa94-4e24-970b-33382e745581-console-serving-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:24:30.584837 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.584638    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz58l\" (UniqueName: \"kubernetes.io/projected/b294ba27-fa94-4e24-970b-33382e745581-kube-api-access-lz58l\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:24:30.584837 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:30.584647    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b294ba27-fa94-4e24-970b-33382e745581-oauth-serving-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:24:31.212771 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:31.212741    2563 generic.go:356] "Generic (PLEG): container finished" podID="b294ba27-fa94-4e24-970b-33382e745581" containerID="2d7038ff0807cc46dca71d033f42f270ed08723b0a3c8a0f2f8618449c8a973e" exitCode=0
Jul 14 15:24:31.213143 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:31.212785    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5b8cc457-ppbqm"
Jul 14 15:24:31.213143 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:31.212830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5b8cc457-ppbqm" event={"ID":"b294ba27-fa94-4e24-970b-33382e745581","Type":"ContainerDied","Data":"2d7038ff0807cc46dca71d033f42f270ed08723b0a3c8a0f2f8618449c8a973e"}
Jul 14 15:24:31.213143 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:31.212875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5b8cc457-ppbqm" event={"ID":"b294ba27-fa94-4e24-970b-33382e745581","Type":"ContainerDied","Data":"b916401e72ee7fcc18c2e4306bd13eb4ab75a5e5b6fad82819f9ec839cc94138"}
Jul 14 15:24:31.213143 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:31.212894    2563 scope.go:117] "RemoveContainer" containerID="2d7038ff0807cc46dca71d033f42f270ed08723b0a3c8a0f2f8618449c8a973e"
Jul 14 15:24:31.221719 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:31.221704    2563 scope.go:117] "RemoveContainer" containerID="2d7038ff0807cc46dca71d033f42f270ed08723b0a3c8a0f2f8618449c8a973e"
Jul 14 15:24:31.221965 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:24:31.221950    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7038ff0807cc46dca71d033f42f270ed08723b0a3c8a0f2f8618449c8a973e\": container with ID starting with 2d7038ff0807cc46dca71d033f42f270ed08723b0a3c8a0f2f8618449c8a973e not found: ID does not exist" containerID="2d7038ff0807cc46dca71d033f42f270ed08723b0a3c8a0f2f8618449c8a973e"
Jul 14 15:24:31.222008 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:31.221971    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7038ff0807cc46dca71d033f42f270ed08723b0a3c8a0f2f8618449c8a973e"} err="failed to get container status \"2d7038ff0807cc46dca71d033f42f270ed08723b0a3c8a0f2f8618449c8a973e\": rpc error: code = NotFound desc = could not find container \"2d7038ff0807cc46dca71d033f42f270ed08723b0a3c8a0f2f8618449c8a973e\": container with ID starting with 2d7038ff0807cc46dca71d033f42f270ed08723b0a3c8a0f2f8618449c8a973e not found: ID does not exist"
Jul 14 15:24:31.238047 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:31.238018    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f5b8cc457-ppbqm"]
Jul 14 15:24:31.247299 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:31.247277    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f5b8cc457-ppbqm"]
Jul 14 15:24:32.215649 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:32.215608    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b294ba27-fa94-4e24-970b-33382e745581" path="/var/lib/kubelet/pods/b294ba27-fa94-4e24-970b-33382e745581/volumes"
Jul 14 15:24:41.256678 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.256625    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-695f5bb85d-hbmjs" podUID="03cf902c-00e1-4e9e-aa4b-ee80f438e2a6" containerName="console" containerID="cri-o://e873caf855e29bc2f48916793c3eab296646f36388d5e1f1ff4d2368779253bb" gracePeriod=15
Jul 14 15:24:41.500134 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.500111    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:24:41.587354 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.587253    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj4vq\" (UniqueName: \"kubernetes.io/projected/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-kube-api-access-xj4vq\") pod \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") "
Jul 14 15:24:41.587354 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.587308    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-service-ca\") pod \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") "
Jul 14 15:24:41.587354 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.587336    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-oauth-serving-cert\") pod \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") "
Jul 14 15:24:41.587663 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.587364    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-serving-cert\") pod \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") "
Jul 14 15:24:41.587663 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.587401    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-oauth-config\") pod \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") "
Jul 14 15:24:41.587663 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.587531    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-config\") pod \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") "
Jul 14 15:24:41.587663 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.587565    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-trusted-ca-bundle\") pod \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\" (UID: \"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6\") "
Jul 14 15:24:41.587859 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.587829    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-service-ca" (OuterVolumeSpecName: "service-ca") pod "03cf902c-00e1-4e9e-aa4b-ee80f438e2a6" (UID: "03cf902c-00e1-4e9e-aa4b-ee80f438e2a6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:24:41.587859 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.587838    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "03cf902c-00e1-4e9e-aa4b-ee80f438e2a6" (UID: "03cf902c-00e1-4e9e-aa4b-ee80f438e2a6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:24:41.588208 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.588187    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "03cf902c-00e1-4e9e-aa4b-ee80f438e2a6" (UID: "03cf902c-00e1-4e9e-aa4b-ee80f438e2a6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:24:41.588298 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.587956    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-config" (OuterVolumeSpecName: "console-config") pod "03cf902c-00e1-4e9e-aa4b-ee80f438e2a6" (UID: "03cf902c-00e1-4e9e-aa4b-ee80f438e2a6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:24:41.589662 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.589630    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "03cf902c-00e1-4e9e-aa4b-ee80f438e2a6" (UID: "03cf902c-00e1-4e9e-aa4b-ee80f438e2a6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:24:41.589758 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.589666    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "03cf902c-00e1-4e9e-aa4b-ee80f438e2a6" (UID: "03cf902c-00e1-4e9e-aa4b-ee80f438e2a6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:24:41.589758 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.589708    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-kube-api-access-xj4vq" (OuterVolumeSpecName: "kube-api-access-xj4vq") pod "03cf902c-00e1-4e9e-aa4b-ee80f438e2a6" (UID: "03cf902c-00e1-4e9e-aa4b-ee80f438e2a6"). InnerVolumeSpecName "kube-api-access-xj4vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:24:41.688277 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.688245    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-oauth-config\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:24:41.688277 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.688273    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-config\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:24:41.688277 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.688282    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-trusted-ca-bundle\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:24:41.688535 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.688291    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj4vq\" (UniqueName: \"kubernetes.io/projected/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-kube-api-access-xj4vq\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:24:41.688535 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.688300    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:24:41.688535 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.688307    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-oauth-serving-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:24:41.688535 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:41.688316    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6-console-serving-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:24:42.255239 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:42.255203    2563 generic.go:356] "Generic (PLEG): container finished" podID="03cf902c-00e1-4e9e-aa4b-ee80f438e2a6" containerID="e873caf855e29bc2f48916793c3eab296646f36388d5e1f1ff4d2368779253bb" exitCode=0
Jul 14 15:24:42.255396 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:42.255250    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695f5bb85d-hbmjs"
Jul 14 15:24:42.255396 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:42.255289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695f5bb85d-hbmjs" event={"ID":"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6","Type":"ContainerDied","Data":"e873caf855e29bc2f48916793c3eab296646f36388d5e1f1ff4d2368779253bb"}
Jul 14 15:24:42.255396 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:42.255342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695f5bb85d-hbmjs" event={"ID":"03cf902c-00e1-4e9e-aa4b-ee80f438e2a6","Type":"ContainerDied","Data":"a8013b09cdf57f7dcc10fb52fa726fd8e5c78d20eca474ba521c6bbbf9b964ca"}
Jul 14 15:24:42.255396 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:42.255363    2563 scope.go:117] "RemoveContainer" containerID="e873caf855e29bc2f48916793c3eab296646f36388d5e1f1ff4d2368779253bb"
Jul 14 15:24:42.263845 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:42.263650    2563 scope.go:117] "RemoveContainer" containerID="e873caf855e29bc2f48916793c3eab296646f36388d5e1f1ff4d2368779253bb"
Jul 14 15:24:42.264056 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:24:42.263900    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e873caf855e29bc2f48916793c3eab296646f36388d5e1f1ff4d2368779253bb\": container with ID starting with e873caf855e29bc2f48916793c3eab296646f36388d5e1f1ff4d2368779253bb not found: ID does not exist" containerID="e873caf855e29bc2f48916793c3eab296646f36388d5e1f1ff4d2368779253bb"
Jul 14 15:24:42.264056 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:42.263925    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e873caf855e29bc2f48916793c3eab296646f36388d5e1f1ff4d2368779253bb"} err="failed to get container status \"e873caf855e29bc2f48916793c3eab296646f36388d5e1f1ff4d2368779253bb\": rpc error: code = NotFound desc = could not find container \"e873caf855e29bc2f48916793c3eab296646f36388d5e1f1ff4d2368779253bb\": container with ID starting with e873caf855e29bc2f48916793c3eab296646f36388d5e1f1ff4d2368779253bb not found: ID does not exist"
Jul 14 15:24:42.273325 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:42.273300    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-695f5bb85d-hbmjs"]
Jul 14 15:24:42.274666 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:42.274649    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-695f5bb85d-hbmjs"]
Jul 14 15:24:44.215030 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:24:44.214999    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03cf902c-00e1-4e9e-aa4b-ee80f438e2a6" path="/var/lib/kubelet/pods/03cf902c-00e1-4e9e-aa4b-ee80f438e2a6/volumes"
Jul 14 15:25:31.799949 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:31.799910    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pbhpn"]
Jul 14 15:25:31.800446 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:31.800419    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b294ba27-fa94-4e24-970b-33382e745581" containerName="console"
Jul 14 15:25:31.800446 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:31.800437    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b294ba27-fa94-4e24-970b-33382e745581" containerName="console"
Jul 14 15:25:31.800598 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:31.800478    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03cf902c-00e1-4e9e-aa4b-ee80f438e2a6" containerName="console"
Jul 14 15:25:31.800598 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:31.800487    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="03cf902c-00e1-4e9e-aa4b-ee80f438e2a6" containerName="console"
Jul 14 15:25:31.800598 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:31.800589    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b294ba27-fa94-4e24-970b-33382e745581" containerName="console"
Jul 14 15:25:31.800737 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:31.800604    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="03cf902c-00e1-4e9e-aa4b-ee80f438e2a6" containerName="console"
Jul 14 15:25:31.805047 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:31.805027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pbhpn"
Jul 14 15:25:31.807752 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:31.807732    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 14 15:25:31.807872 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:31.807736    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5k6bl\""
Jul 14 15:25:31.809341 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:31.809319    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 14 15:25:31.810980 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:31.810951    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pbhpn"]
Jul 14 15:25:31.934126 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:31.934094    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cd64ddc-180d-40f9-8c34-46fb05812e3c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pbhpn\" (UID: \"1cd64ddc-180d-40f9-8c34-46fb05812e3c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pbhpn"
Jul 14 15:25:31.934126 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:31.934132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdpvt\" (UniqueName: \"kubernetes.io/projected/1cd64ddc-180d-40f9-8c34-46fb05812e3c-kube-api-access-rdpvt\") pod \"cert-manager-cainjector-559cdd8c87-pbhpn\" (UID: \"1cd64ddc-180d-40f9-8c34-46fb05812e3c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pbhpn"
Jul 14 15:25:32.034800 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:32.034768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cd64ddc-180d-40f9-8c34-46fb05812e3c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pbhpn\" (UID: \"1cd64ddc-180d-40f9-8c34-46fb05812e3c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pbhpn"
Jul 14 15:25:32.034978 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:32.034820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdpvt\" (UniqueName: \"kubernetes.io/projected/1cd64ddc-180d-40f9-8c34-46fb05812e3c-kube-api-access-rdpvt\") pod \"cert-manager-cainjector-559cdd8c87-pbhpn\" (UID: \"1cd64ddc-180d-40f9-8c34-46fb05812e3c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pbhpn"
Jul 14 15:25:32.043523 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:32.043467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cd64ddc-180d-40f9-8c34-46fb05812e3c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pbhpn\" (UID: \"1cd64ddc-180d-40f9-8c34-46fb05812e3c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pbhpn"
Jul 14 15:25:32.043722 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:32.043704    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdpvt\" (UniqueName: \"kubernetes.io/projected/1cd64ddc-180d-40f9-8c34-46fb05812e3c-kube-api-access-rdpvt\") pod \"cert-manager-cainjector-559cdd8c87-pbhpn\" (UID: \"1cd64ddc-180d-40f9-8c34-46fb05812e3c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pbhpn"
Jul 14 15:25:32.128458 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:32.128367    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pbhpn"
Jul 14 15:25:32.256469 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:32.256447    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pbhpn"]
Jul 14 15:25:32.259166 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:25:32.259118    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cd64ddc_180d_40f9_8c34_46fb05812e3c.slice/crio-7d6ed733336169377d09bd9028791ddbf465d31382332b18713fc2d8ea20a9b3 WatchSource:0}: Error finding container 7d6ed733336169377d09bd9028791ddbf465d31382332b18713fc2d8ea20a9b3: Status 404 returned error can't find the container with id 7d6ed733336169377d09bd9028791ddbf465d31382332b18713fc2d8ea20a9b3
Jul 14 15:25:32.461113 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:32.461081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pbhpn" event={"ID":"1cd64ddc-180d-40f9-8c34-46fb05812e3c","Type":"ContainerStarted","Data":"7d6ed733336169377d09bd9028791ddbf465d31382332b18713fc2d8ea20a9b3"}
Jul 14 15:25:35.475849 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:35.475813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pbhpn" event={"ID":"1cd64ddc-180d-40f9-8c34-46fb05812e3c","Type":"ContainerStarted","Data":"9ce9e67b37bb2caf7f75332c97a9d2cfcb6be16df60401c1b6f7190258e3a4e6"}
Jul 14 15:25:35.496010 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:35.495952    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pbhpn" podStartSLOduration=1.6532733450000001 podStartE2EDuration="4.495934919s" totalImagesPullingTime="2.842661574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:25:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:25:35.494424511 +0000 UTC m=+259.835690419" watchObservedRunningTime="2026-07-14 15:25:35.495934919 +0000 UTC m=+259.837200825"
Jul 14 15:25:41.029585 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:41.029482    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bwf7h"]
Jul 14 15:25:41.032798 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:41.032779    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bwf7h"
Jul 14 15:25:41.035376 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:41.035357    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-pwstx\""
Jul 14 15:25:41.042710 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:41.042686    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bwf7h"]
Jul 14 15:25:41.217073 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:41.217039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/185644cc-6069-488d-ba5e-2a7917863c2c-bound-sa-token\") pod \"cert-manager-55b4d954d5-bwf7h\" (UID: \"185644cc-6069-488d-ba5e-2a7917863c2c\") " pod="cert-manager/cert-manager-55b4d954d5-bwf7h"
Jul 14 15:25:41.217241 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:41.217092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2mvt\" (UniqueName: \"kubernetes.io/projected/185644cc-6069-488d-ba5e-2a7917863c2c-kube-api-access-h2mvt\") pod \"cert-manager-55b4d954d5-bwf7h\" (UID: \"185644cc-6069-488d-ba5e-2a7917863c2c\") " pod="cert-manager/cert-manager-55b4d954d5-bwf7h"
Jul 14 15:25:41.318104 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:41.318019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/185644cc-6069-488d-ba5e-2a7917863c2c-bound-sa-token\") pod \"cert-manager-55b4d954d5-bwf7h\" (UID: \"185644cc-6069-488d-ba5e-2a7917863c2c\") " pod="cert-manager/cert-manager-55b4d954d5-bwf7h"
Jul 14 15:25:41.318104 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:41.318078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2mvt\" (UniqueName: \"kubernetes.io/projected/185644cc-6069-488d-ba5e-2a7917863c2c-kube-api-access-h2mvt\") pod \"cert-manager-55b4d954d5-bwf7h\" (UID: \"185644cc-6069-488d-ba5e-2a7917863c2c\") " pod="cert-manager/cert-manager-55b4d954d5-bwf7h"
Jul 14 15:25:41.326811 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:41.326777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/185644cc-6069-488d-ba5e-2a7917863c2c-bound-sa-token\") pod \"cert-manager-55b4d954d5-bwf7h\" (UID: \"185644cc-6069-488d-ba5e-2a7917863c2c\") " pod="cert-manager/cert-manager-55b4d954d5-bwf7h"
Jul 14 15:25:41.326930 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:41.326816    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2mvt\" (UniqueName: \"kubernetes.io/projected/185644cc-6069-488d-ba5e-2a7917863c2c-kube-api-access-h2mvt\") pod \"cert-manager-55b4d954d5-bwf7h\" (UID: \"185644cc-6069-488d-ba5e-2a7917863c2c\") " pod="cert-manager/cert-manager-55b4d954d5-bwf7h"
Jul 14 15:25:41.341865 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:41.341837    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bwf7h"
Jul 14 15:25:41.490251 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:41.490229    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bwf7h"]
Jul 14 15:25:41.493629 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:25:41.493595    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod185644cc_6069_488d_ba5e_2a7917863c2c.slice/crio-762b88f35724b61cd0eb05cc42592c942bf611dac076781612caf0ff5aea4bc9 WatchSource:0}: Error finding container 762b88f35724b61cd0eb05cc42592c942bf611dac076781612caf0ff5aea4bc9: Status 404 returned error can't find the container with id 762b88f35724b61cd0eb05cc42592c942bf611dac076781612caf0ff5aea4bc9
Jul 14 15:25:41.508228 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:41.508192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bwf7h" event={"ID":"185644cc-6069-488d-ba5e-2a7917863c2c","Type":"ContainerStarted","Data":"762b88f35724b61cd0eb05cc42592c942bf611dac076781612caf0ff5aea4bc9"}
Jul 14 15:25:42.513199 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:42.513160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bwf7h" event={"ID":"185644cc-6069-488d-ba5e-2a7917863c2c","Type":"ContainerStarted","Data":"8e17e70cf612856bbb946d211b6b467e330309f8193cb19464b747890095ebd4"}
Jul 14 15:25:42.530151 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:25:42.530106    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bwf7h" podStartSLOduration=1.530092025 podStartE2EDuration="1.530092025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:25:42.529694555 +0000 UTC m=+266.870960462" watchObservedRunningTime="2026-07-14 15:25:42.530092025 +0000 UTC m=+266.871357931"
Jul 14 15:26:04.750339 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.750306    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"]
Jul 14 15:26:04.754006 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.753989    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.757648 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.757625    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 14 15:26:04.759399 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.759380    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 14 15:26:04.759530 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.759496    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 14 15:26:04.760332 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.760307    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-765p8\""
Jul 14 15:26:04.760430 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.760360    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 14 15:26:04.765151 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.765127    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"]
Jul 14 15:26:04.826538 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.826469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e8f4479-2e26-46c8-9b6d-45f72af91d02-manifests\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.826721 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.826543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bc92\" (UniqueName: \"kubernetes.io/projected/3e8f4479-2e26-46c8-9b6d-45f72af91d02-kube-api-access-5bc92\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.826721 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.826598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3e8f4479-2e26-46c8-9b6d-45f72af91d02-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.826721 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.826673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e8f4479-2e26-46c8-9b6d-45f72af91d02-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.826845 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.826747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e8f4479-2e26-46c8-9b6d-45f72af91d02-tmp\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.927556 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.927514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e8f4479-2e26-46c8-9b6d-45f72af91d02-tmp\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.927754 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.927574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e8f4479-2e26-46c8-9b6d-45f72af91d02-manifests\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.927754 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.927604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bc92\" (UniqueName: \"kubernetes.io/projected/3e8f4479-2e26-46c8-9b6d-45f72af91d02-kube-api-access-5bc92\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.927754 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.927650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3e8f4479-2e26-46c8-9b6d-45f72af91d02-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.927754 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.927686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e8f4479-2e26-46c8-9b6d-45f72af91d02-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.928085 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.928053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e8f4479-2e26-46c8-9b6d-45f72af91d02-manifests\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.929902 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.929881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e8f4479-2e26-46c8-9b6d-45f72af91d02-tmp\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.930097 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.930082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e8f4479-2e26-46c8-9b6d-45f72af91d02-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.930137 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.930118    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3e8f4479-2e26-46c8-9b6d-45f72af91d02-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:04.938170 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:04.938142    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bc92\" (UniqueName: \"kubernetes.io/projected/3e8f4479-2e26-46c8-9b6d-45f72af91d02-kube-api-access-5bc92\") pod \"opendatahub-operator-controller-manager-66c4b7dd84-7j48t\" (UID: \"3e8f4479-2e26-46c8-9b6d-45f72af91d02\") " pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:05.065624 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:05.065499    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:05.205449 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:05.205427    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"]
Jul 14 15:26:05.207831 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:26:05.207808    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8f4479_2e26_46c8_9b6d_45f72af91d02.slice/crio-3f2988b4c0235b658537bf8742b5904234caa21ecb8799579dbc570a6667921d WatchSource:0}: Error finding container 3f2988b4c0235b658537bf8742b5904234caa21ecb8799579dbc570a6667921d: Status 404 returned error can't find the container with id 3f2988b4c0235b658537bf8742b5904234caa21ecb8799579dbc570a6667921d
Jul 14 15:26:05.607032 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:05.606996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t" event={"ID":"3e8f4479-2e26-46c8-9b6d-45f72af91d02","Type":"ContainerStarted","Data":"3f2988b4c0235b658537bf8742b5904234caa21ecb8799579dbc570a6667921d"}
Jul 14 15:26:10.634099 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:10.634064    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t" event={"ID":"3e8f4479-2e26-46c8-9b6d-45f72af91d02","Type":"ContainerStarted","Data":"f176da7eea7026d578daf6698e55348e704f719138f7182ada98fe50f375e8a2"}
Jul 14 15:26:11.640069 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:11.640033    2563 generic.go:356] "Generic (PLEG): container finished" podID="3e8f4479-2e26-46c8-9b6d-45f72af91d02" containerID="f176da7eea7026d578daf6698e55348e704f719138f7182ada98fe50f375e8a2" exitCode=0
Jul 14 15:26:11.640539 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:11.640079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t" event={"ID":"3e8f4479-2e26-46c8-9b6d-45f72af91d02","Type":"ContainerDied","Data":"f176da7eea7026d578daf6698e55348e704f719138f7182ada98fe50f375e8a2"}
Jul 14 15:26:12.646659 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:12.646626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t" event={"ID":"3e8f4479-2e26-46c8-9b6d-45f72af91d02","Type":"ContainerStarted","Data":"46253c2f7a2905c9de81b58407ea583b4ec6706bc5909a5d62fa125e36a6e5df"}
Jul 14 15:26:12.647119 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:12.646752    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:12.672590 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:12.672542    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t" podStartSLOduration=1.921527476 podStartE2EDuration="8.672527253s" totalImagesPullingTime="5.75209439s" totalInitContainerRuntime="998.905387ms" isStatefulPod=false podCreationTimestamp="2026-07-14 15:26:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:26:12.67179269 +0000 UTC m=+297.013058617" watchObservedRunningTime="2026-07-14 15:26:12.672527253 +0000 UTC m=+297.013793154"
Jul 14 15:26:16.028797 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:16.028766    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:26:16.031318 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:16.031297    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:26:16.035478 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:16.035450    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 14 15:26:23.655414 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:23.655384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-66c4b7dd84-7j48t"
Jul 14 15:26:37.903291 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:37.903258    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wg277"]
Jul 14 15:26:37.908245 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:37.908222    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc"]
Jul 14 15:26:37.908376 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:37.908335    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wg277"
Jul 14 15:26:37.911978 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:37.911958    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc"
Jul 14 15:26:37.912525 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:37.912484    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8zsxb\""
Jul 14 15:26:37.912657 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:37.912564    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 14 15:26:37.914539 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:37.914519    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-6qzgr\""
Jul 14 15:26:37.914647 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:37.914517    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 14 15:26:37.923992 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:37.923966    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wg277"]
Jul 14 15:26:37.925985 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:37.925962    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc"]
Jul 14 15:26:37.931244 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:37.931221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr8m7\" (UniqueName: \"kubernetes.io/projected/0ccbedea-ad39-458c-89af-4fbb1801359a-kube-api-access-xr8m7\") pod \"kserve-controller-manager-d6f4b65b8-wg277\" (UID: \"0ccbedea-ad39-458c-89af-4fbb1801359a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wg277"
Jul 14 15:26:37.931350 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:37.931258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ccbedea-ad39-458c-89af-4fbb1801359a-cert\") pod \"kserve-controller-manager-d6f4b65b8-wg277\" (UID: \"0ccbedea-ad39-458c-89af-4fbb1801359a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wg277"
Jul 14 15:26:37.998298 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:37.998264    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"]
Jul 14 15:26:38.001578 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.001556    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:38.004552 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.004532    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 14 15:26:38.004681 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.004556    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9dwkj\""
Jul 14 15:26:38.004681 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.004538    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 14 15:26:38.004681 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.004536    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 14 15:26:38.004681 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.004540    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 14 15:26:38.004681 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.004537    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 14 15:26:38.010887 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.010865    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"]
Jul 14 15:26:38.031857 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.031824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvksk\" (UniqueName: \"kubernetes.io/projected/1367dbde-4785-4189-a4e5-cb457c93c18b-kube-api-access-xvksk\") pod \"llmisvc-controller-manager-777c6bd7d5-wsnwc\" (UID: \"1367dbde-4785-4189-a4e5-cb457c93c18b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc"
Jul 14 15:26:38.032027 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.031878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr8m7\" (UniqueName: \"kubernetes.io/projected/0ccbedea-ad39-458c-89af-4fbb1801359a-kube-api-access-xr8m7\") pod \"kserve-controller-manager-d6f4b65b8-wg277\" (UID: \"0ccbedea-ad39-458c-89af-4fbb1801359a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wg277"
Jul 14 15:26:38.032027 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.032019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1367dbde-4785-4189-a4e5-cb457c93c18b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wsnwc\" (UID: \"1367dbde-4785-4189-a4e5-cb457c93c18b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc"
Jul 14 15:26:38.032144 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.032048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ccbedea-ad39-458c-89af-4fbb1801359a-cert\") pod \"kserve-controller-manager-d6f4b65b8-wg277\" (UID: \"0ccbedea-ad39-458c-89af-4fbb1801359a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wg277"
Jul 14 15:26:38.032144 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.032119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbmp6\" (UniqueName: \"kubernetes.io/projected/d8318403-37b9-4676-95b1-4ea50e076a0d-kube-api-access-dbmp6\") pod \"lws-controller-manager-d87d6d9c6-ncbx6\" (UID: \"d8318403-37b9-4676-95b1-4ea50e076a0d\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:38.032217 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.032148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d8318403-37b9-4676-95b1-4ea50e076a0d-manager-config\") pod \"lws-controller-manager-d87d6d9c6-ncbx6\" (UID: \"d8318403-37b9-4676-95b1-4ea50e076a0d\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:38.032254 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.032235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8318403-37b9-4676-95b1-4ea50e076a0d-cert\") pod \"lws-controller-manager-d87d6d9c6-ncbx6\" (UID: \"d8318403-37b9-4676-95b1-4ea50e076a0d\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:38.032296 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.032281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8318403-37b9-4676-95b1-4ea50e076a0d-metrics-cert\") pod \"lws-controller-manager-d87d6d9c6-ncbx6\" (UID: \"d8318403-37b9-4676-95b1-4ea50e076a0d\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:38.034548 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.034528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ccbedea-ad39-458c-89af-4fbb1801359a-cert\") pod \"kserve-controller-manager-d6f4b65b8-wg277\" (UID: \"0ccbedea-ad39-458c-89af-4fbb1801359a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wg277"
Jul 14 15:26:38.045559 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.045534    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr8m7\" (UniqueName: \"kubernetes.io/projected/0ccbedea-ad39-458c-89af-4fbb1801359a-kube-api-access-xr8m7\") pod \"kserve-controller-manager-d6f4b65b8-wg277\" (UID: \"0ccbedea-ad39-458c-89af-4fbb1801359a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wg277"
Jul 14 15:26:38.132829 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.132795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbmp6\" (UniqueName: \"kubernetes.io/projected/d8318403-37b9-4676-95b1-4ea50e076a0d-kube-api-access-dbmp6\") pod \"lws-controller-manager-d87d6d9c6-ncbx6\" (UID: \"d8318403-37b9-4676-95b1-4ea50e076a0d\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:38.132829 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.132832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d8318403-37b9-4676-95b1-4ea50e076a0d-manager-config\") pod \"lws-controller-manager-d87d6d9c6-ncbx6\" (UID: \"d8318403-37b9-4676-95b1-4ea50e076a0d\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:38.133049 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.132885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8318403-37b9-4676-95b1-4ea50e076a0d-cert\") pod \"lws-controller-manager-d87d6d9c6-ncbx6\" (UID: \"d8318403-37b9-4676-95b1-4ea50e076a0d\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:38.133049 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.132912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8318403-37b9-4676-95b1-4ea50e076a0d-metrics-cert\") pod \"lws-controller-manager-d87d6d9c6-ncbx6\" (UID: \"d8318403-37b9-4676-95b1-4ea50e076a0d\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:38.133049 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.132935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvksk\" (UniqueName: \"kubernetes.io/projected/1367dbde-4785-4189-a4e5-cb457c93c18b-kube-api-access-xvksk\") pod \"llmisvc-controller-manager-777c6bd7d5-wsnwc\" (UID: \"1367dbde-4785-4189-a4e5-cb457c93c18b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc"
Jul 14 15:26:38.133049 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.132973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1367dbde-4785-4189-a4e5-cb457c93c18b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wsnwc\" (UID: \"1367dbde-4785-4189-a4e5-cb457c93c18b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc"
Jul 14 15:26:38.133243 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:26:38.133064    2563 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 14 15:26:38.133243 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:26:38.133116    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1367dbde-4785-4189-a4e5-cb457c93c18b-cert podName:1367dbde-4785-4189-a4e5-cb457c93c18b nodeName:}" failed. No retries permitted until 2026-07-14 15:26:38.633096483 +0000 UTC m=+322.974362368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1367dbde-4785-4189-a4e5-cb457c93c18b-cert") pod "llmisvc-controller-manager-777c6bd7d5-wsnwc" (UID: "1367dbde-4785-4189-a4e5-cb457c93c18b") : secret "llmisvc-webhook-server-cert" not found
Jul 14 15:26:38.133758 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.133691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d8318403-37b9-4676-95b1-4ea50e076a0d-manager-config\") pod \"lws-controller-manager-d87d6d9c6-ncbx6\" (UID: \"d8318403-37b9-4676-95b1-4ea50e076a0d\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:38.135462 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.135435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8318403-37b9-4676-95b1-4ea50e076a0d-metrics-cert\") pod \"lws-controller-manager-d87d6d9c6-ncbx6\" (UID: \"d8318403-37b9-4676-95b1-4ea50e076a0d\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:38.135601 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.135582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8318403-37b9-4676-95b1-4ea50e076a0d-cert\") pod \"lws-controller-manager-d87d6d9c6-ncbx6\" (UID: \"d8318403-37b9-4676-95b1-4ea50e076a0d\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:38.143841 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.143822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbmp6\" (UniqueName: \"kubernetes.io/projected/d8318403-37b9-4676-95b1-4ea50e076a0d-kube-api-access-dbmp6\") pod \"lws-controller-manager-d87d6d9c6-ncbx6\" (UID: \"d8318403-37b9-4676-95b1-4ea50e076a0d\") " pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:38.146873 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.146855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvksk\" (UniqueName: \"kubernetes.io/projected/1367dbde-4785-4189-a4e5-cb457c93c18b-kube-api-access-xvksk\") pod \"llmisvc-controller-manager-777c6bd7d5-wsnwc\" (UID: \"1367dbde-4785-4189-a4e5-cb457c93c18b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc"
Jul 14 15:26:38.220538 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.220432    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wg277"
Jul 14 15:26:38.311883 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.311852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:38.356643 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.356577    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wg277"]
Jul 14 15:26:38.358938 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:26:38.358910    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ccbedea_ad39_458c_89af_4fbb1801359a.slice/crio-7c4f32c39c53eb3c327777a02ac03b02d749e89c8e09cdb01bf795eb71c9e49a WatchSource:0}: Error finding container 7c4f32c39c53eb3c327777a02ac03b02d749e89c8e09cdb01bf795eb71c9e49a: Status 404 returned error can't find the container with id 7c4f32c39c53eb3c327777a02ac03b02d749e89c8e09cdb01bf795eb71c9e49a
Jul 14 15:26:38.360575 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.360558    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 15:26:38.468306 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.468145    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"]
Jul 14 15:26:38.471570 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:26:38.471491    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8318403_37b9_4676_95b1_4ea50e076a0d.slice/crio-cd6036833abd7f64c0df11a0447577c7a09f7615ea7c873c7d3fbc14a3d5daee WatchSource:0}: Error finding container cd6036833abd7f64c0df11a0447577c7a09f7615ea7c873c7d3fbc14a3d5daee: Status 404 returned error can't find the container with id cd6036833abd7f64c0df11a0447577c7a09f7615ea7c873c7d3fbc14a3d5daee
Jul 14 15:26:38.637145 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.637116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1367dbde-4785-4189-a4e5-cb457c93c18b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wsnwc\" (UID: \"1367dbde-4785-4189-a4e5-cb457c93c18b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc"
Jul 14 15:26:38.639767 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.639738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1367dbde-4785-4189-a4e5-cb457c93c18b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wsnwc\" (UID: \"1367dbde-4785-4189-a4e5-cb457c93c18b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc"
Jul 14 15:26:38.773857 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.773770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6" event={"ID":"d8318403-37b9-4676-95b1-4ea50e076a0d","Type":"ContainerStarted","Data":"cd6036833abd7f64c0df11a0447577c7a09f7615ea7c873c7d3fbc14a3d5daee"}
Jul 14 15:26:38.774779 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.774760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wg277" event={"ID":"0ccbedea-ad39-458c-89af-4fbb1801359a","Type":"ContainerStarted","Data":"7c4f32c39c53eb3c327777a02ac03b02d749e89c8e09cdb01bf795eb71c9e49a"}
Jul 14 15:26:38.827283 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.827247    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc"
Jul 14 15:26:38.996777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:38.996748    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc"]
Jul 14 15:26:39.002328 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:26:39.002296    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1367dbde_4785_4189_a4e5_cb457c93c18b.slice/crio-22c9def199fd4841f04584266253958ce76f23dd3fd350018a24ce33be2e450a WatchSource:0}: Error finding container 22c9def199fd4841f04584266253958ce76f23dd3fd350018a24ce33be2e450a: Status 404 returned error can't find the container with id 22c9def199fd4841f04584266253958ce76f23dd3fd350018a24ce33be2e450a
Jul 14 15:26:39.782454 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:39.782419    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc" event={"ID":"1367dbde-4785-4189-a4e5-cb457c93c18b","Type":"ContainerStarted","Data":"22c9def199fd4841f04584266253958ce76f23dd3fd350018a24ce33be2e450a"}
Jul 14 15:26:41.797531 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:41.797414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wg277" event={"ID":"0ccbedea-ad39-458c-89af-4fbb1801359a","Type":"ContainerStarted","Data":"fedc595fd1c0ef4b3959988c942a55f65cf8c581fe8bb6d6b3f2bc42b1c13442"}
Jul 14 15:26:41.797531 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:41.797492    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wg277"
Jul 14 15:26:41.830672 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:41.830615    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wg277" podStartSLOduration=1.884365418 podStartE2EDuration="4.830594464s" totalImagesPullingTime="2.946229046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:26:41.829679055 +0000 UTC m=+326.170944981" watchObservedRunningTime="2026-07-14 15:26:41.830594464 +0000 UTC m=+326.171860412"
Jul 14 15:26:44.816241 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:44.816212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc" event={"ID":"1367dbde-4785-4189-a4e5-cb457c93c18b","Type":"ContainerStarted","Data":"3021104b7206e58b697ef91b0b38545a787ac6b438e6b761c351cab5ba21c25e"}
Jul 14 15:26:44.816707 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:44.816320    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc"
Jul 14 15:26:44.817739 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:44.817718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6" event={"ID":"d8318403-37b9-4676-95b1-4ea50e076a0d","Type":"ContainerStarted","Data":"51520a5ea263b3bc528ce5f3b06895190ad0eef2cf12778ba67b7a23f674e90d"}
Jul 14 15:26:44.817876 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:44.817861    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:26:44.833364 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:44.833313    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc" podStartSLOduration=2.705655261 podStartE2EDuration="7.833298821s" totalImagesPullingTime="5.12764356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:26:44.831777935 +0000 UTC m=+329.173043862" watchObservedRunningTime="2026-07-14 15:26:44.833298821 +0000 UTC m=+329.174564706"
Jul 14 15:26:44.849000 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:44.848953    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6" podStartSLOduration=2.133492652 podStartE2EDuration="7.848936361s" totalImagesPullingTime="5.715443709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:26:44.847561092 +0000 UTC m=+329.188827030" watchObservedRunningTime="2026-07-14 15:26:44.848936361 +0000 UTC m=+329.190202334"
Jul 14 15:26:55.825175 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:26:55.825146    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d87d6d9c6-ncbx6"
Jul 14 15:27:12.808853 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:12.808787    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wg277"
Jul 14 15:27:15.824859 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:15.824833    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wsnwc"
Jul 14 15:27:28.659478 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:28.659441    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s6xtj"]
Jul 14 15:27:28.666107 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:28.666083    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-s6xtj"
Jul 14 15:27:28.668805 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:28.668780    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 14 15:27:28.668963 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:28.668948    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-npxw7\""
Jul 14 15:27:28.670386 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:28.670360    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 14 15:27:28.674666 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:28.674235    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s6xtj"]
Jul 14 15:27:28.792766 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:28.792732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vddzt\" (UniqueName: \"kubernetes.io/projected/734be19f-25d9-4b17-aa58-220962241fae-kube-api-access-vddzt\") pod \"kuadrant-operator-catalog-s6xtj\" (UID: \"734be19f-25d9-4b17-aa58-220962241fae\") " pod="kuadrant-system/kuadrant-operator-catalog-s6xtj"
Jul 14 15:27:28.893446 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:28.893412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vddzt\" (UniqueName: \"kubernetes.io/projected/734be19f-25d9-4b17-aa58-220962241fae-kube-api-access-vddzt\") pod \"kuadrant-operator-catalog-s6xtj\" (UID: \"734be19f-25d9-4b17-aa58-220962241fae\") " pod="kuadrant-system/kuadrant-operator-catalog-s6xtj"
Jul 14 15:27:28.904571 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:28.904544    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vddzt\" (UniqueName: \"kubernetes.io/projected/734be19f-25d9-4b17-aa58-220962241fae-kube-api-access-vddzt\") pod \"kuadrant-operator-catalog-s6xtj\" (UID: \"734be19f-25d9-4b17-aa58-220962241fae\") " pod="kuadrant-system/kuadrant-operator-catalog-s6xtj"
Jul 14 15:27:28.977451 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:28.977364    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-s6xtj"
Jul 14 15:27:29.109985 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:29.109962    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s6xtj"]
Jul 14 15:27:29.112313 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:27:29.112287    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734be19f_25d9_4b17_aa58_220962241fae.slice/crio-30bebaeb050c392972822c272289e0497d8dc8aefd16d827276e5200c25713a2 WatchSource:0}: Error finding container 30bebaeb050c392972822c272289e0497d8dc8aefd16d827276e5200c25713a2: Status 404 returned error can't find the container with id 30bebaeb050c392972822c272289e0497d8dc8aefd16d827276e5200c25713a2
Jul 14 15:27:29.338766 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:29.338683    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s6xtj"]
Jul 14 15:27:29.355544 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:29.355514    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wvm95"]
Jul 14 15:27:29.358493 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:29.358477    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wvm95"
Jul 14 15:27:29.365898 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:29.365877    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wvm95"]
Jul 14 15:27:29.398984 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:29.398948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d267q\" (UniqueName: \"kubernetes.io/projected/81a4323c-6cfb-4f75-8091-9d4fc9d2ac70-kube-api-access-d267q\") pod \"kuadrant-operator-catalog-wvm95\" (UID: \"81a4323c-6cfb-4f75-8091-9d4fc9d2ac70\") " pod="kuadrant-system/kuadrant-operator-catalog-wvm95"
Jul 14 15:27:29.500057 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:29.500021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d267q\" (UniqueName: \"kubernetes.io/projected/81a4323c-6cfb-4f75-8091-9d4fc9d2ac70-kube-api-access-d267q\") pod \"kuadrant-operator-catalog-wvm95\" (UID: \"81a4323c-6cfb-4f75-8091-9d4fc9d2ac70\") " pod="kuadrant-system/kuadrant-operator-catalog-wvm95"
Jul 14 15:27:29.509281 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:29.509248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d267q\" (UniqueName: \"kubernetes.io/projected/81a4323c-6cfb-4f75-8091-9d4fc9d2ac70-kube-api-access-d267q\") pod \"kuadrant-operator-catalog-wvm95\" (UID: \"81a4323c-6cfb-4f75-8091-9d4fc9d2ac70\") " pod="kuadrant-system/kuadrant-operator-catalog-wvm95"
Jul 14 15:27:29.669106 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:29.669071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wvm95"
Jul 14 15:27:29.801954 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:29.801927    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wvm95"]
Jul 14 15:27:29.804339 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:27:29.804313    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a4323c_6cfb_4f75_8091_9d4fc9d2ac70.slice/crio-07859d3114ee34482893564a244e98db75a2980c59c0ee02341212e1f9e14ac7 WatchSource:0}: Error finding container 07859d3114ee34482893564a244e98db75a2980c59c0ee02341212e1f9e14ac7: Status 404 returned error can't find the container with id 07859d3114ee34482893564a244e98db75a2980c59c0ee02341212e1f9e14ac7
Jul 14 15:27:30.022343 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:30.022259    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wvm95" event={"ID":"81a4323c-6cfb-4f75-8091-9d4fc9d2ac70","Type":"ContainerStarted","Data":"07859d3114ee34482893564a244e98db75a2980c59c0ee02341212e1f9e14ac7"}
Jul 14 15:27:30.023557 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:30.023533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-s6xtj" event={"ID":"734be19f-25d9-4b17-aa58-220962241fae","Type":"ContainerStarted","Data":"30bebaeb050c392972822c272289e0497d8dc8aefd16d827276e5200c25713a2"}
Jul 14 15:27:32.035595 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:32.035554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wvm95" event={"ID":"81a4323c-6cfb-4f75-8091-9d4fc9d2ac70","Type":"ContainerStarted","Data":"3d070934b68d9da334d2e6667a9270c25a3b4abc14ff58faffbde2ab68c21ee0"}
Jul 14 15:27:32.037212 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:32.037189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-s6xtj" event={"ID":"734be19f-25d9-4b17-aa58-220962241fae","Type":"ContainerStarted","Data":"c9b927478f65addb198b00e09b0b5078c8c32eb4d6782ea83cd52392aaaba99e"}
Jul 14 15:27:32.037297 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:32.037242    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-s6xtj" podUID="734be19f-25d9-4b17-aa58-220962241fae" containerName="registry-server" containerID="cri-o://c9b927478f65addb198b00e09b0b5078c8c32eb4d6782ea83cd52392aaaba99e" gracePeriod=2
Jul 14 15:27:32.054126 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:32.054084    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wvm95" podStartSLOduration=1.275625568 podStartE2EDuration="3.054067743s" totalImagesPullingTime="1.778442175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:27:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:27:32.051437613 +0000 UTC m=+376.392703538" watchObservedRunningTime="2026-07-14 15:27:32.054067743 +0000 UTC m=+376.395333649"
Jul 14 15:27:32.069853 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:32.069803    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-s6xtj" podStartSLOduration=1.602282827 podStartE2EDuration="4.069787044s" totalImagesPullingTime="2.467504217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:27:32.069468828 +0000 UTC m=+376.410734753" watchObservedRunningTime="2026-07-14 15:27:32.069787044 +0000 UTC m=+376.411052951"
Jul 14 15:27:32.287363 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:32.287297    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-s6xtj"
Jul 14 15:27:32.326337 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:32.326300    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vddzt\" (UniqueName: \"kubernetes.io/projected/734be19f-25d9-4b17-aa58-220962241fae-kube-api-access-vddzt\") pod \"734be19f-25d9-4b17-aa58-220962241fae\" (UID: \"734be19f-25d9-4b17-aa58-220962241fae\") "
Jul 14 15:27:32.328764 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:32.328729    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734be19f-25d9-4b17-aa58-220962241fae-kube-api-access-vddzt" (OuterVolumeSpecName: "kube-api-access-vddzt") pod "734be19f-25d9-4b17-aa58-220962241fae" (UID: "734be19f-25d9-4b17-aa58-220962241fae"). InnerVolumeSpecName "kube-api-access-vddzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:27:32.427816 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:32.427775    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vddzt\" (UniqueName: \"kubernetes.io/projected/734be19f-25d9-4b17-aa58-220962241fae-kube-api-access-vddzt\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:27:33.043068 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:33.043026    2563 generic.go:356] "Generic (PLEG): container finished" podID="734be19f-25d9-4b17-aa58-220962241fae" containerID="c9b927478f65addb198b00e09b0b5078c8c32eb4d6782ea83cd52392aaaba99e" exitCode=0
Jul 14 15:27:33.043068 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:33.043066    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-s6xtj"
Jul 14 15:27:33.043543 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:33.043114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-s6xtj" event={"ID":"734be19f-25d9-4b17-aa58-220962241fae","Type":"ContainerDied","Data":"c9b927478f65addb198b00e09b0b5078c8c32eb4d6782ea83cd52392aaaba99e"}
Jul 14 15:27:33.043543 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:33.043168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-s6xtj" event={"ID":"734be19f-25d9-4b17-aa58-220962241fae","Type":"ContainerDied","Data":"30bebaeb050c392972822c272289e0497d8dc8aefd16d827276e5200c25713a2"}
Jul 14 15:27:33.043543 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:33.043190    2563 scope.go:117] "RemoveContainer" containerID="c9b927478f65addb198b00e09b0b5078c8c32eb4d6782ea83cd52392aaaba99e"
Jul 14 15:27:33.059466 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:33.059446    2563 scope.go:117] "RemoveContainer" containerID="c9b927478f65addb198b00e09b0b5078c8c32eb4d6782ea83cd52392aaaba99e"
Jul 14 15:27:33.059812 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:27:33.059791    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b927478f65addb198b00e09b0b5078c8c32eb4d6782ea83cd52392aaaba99e\": container with ID starting with c9b927478f65addb198b00e09b0b5078c8c32eb4d6782ea83cd52392aaaba99e not found: ID does not exist" containerID="c9b927478f65addb198b00e09b0b5078c8c32eb4d6782ea83cd52392aaaba99e"
Jul 14 15:27:33.059860 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:33.059820    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b927478f65addb198b00e09b0b5078c8c32eb4d6782ea83cd52392aaaba99e"} err="failed to get container status \"c9b927478f65addb198b00e09b0b5078c8c32eb4d6782ea83cd52392aaaba99e\": rpc error: code = NotFound desc = could not find container \"c9b927478f65addb198b00e09b0b5078c8c32eb4d6782ea83cd52392aaaba99e\": container with ID starting with c9b927478f65addb198b00e09b0b5078c8c32eb4d6782ea83cd52392aaaba99e not found: ID does not exist"
Jul 14 15:27:33.072340 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:33.072311    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s6xtj"]
Jul 14 15:27:33.074304 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:33.074281    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s6xtj"]
Jul 14 15:27:34.215941 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:34.215912    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="734be19f-25d9-4b17-aa58-220962241fae" path="/var/lib/kubelet/pods/734be19f-25d9-4b17-aa58-220962241fae/volumes"
Jul 14 15:27:39.669305 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:39.669263    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-wvm95"
Jul 14 15:27:39.669766 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:39.669336    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-wvm95"
Jul 14 15:27:39.693778 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:39.693751    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-wvm95"
Jul 14 15:27:40.100595 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:27:40.100493    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-wvm95"
Jul 14 15:28:06.405025 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:06.404990    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltthc"]
Jul 14 15:28:06.405453 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:06.405409    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="734be19f-25d9-4b17-aa58-220962241fae" containerName="registry-server"
Jul 14 15:28:06.405453 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:06.405420    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="734be19f-25d9-4b17-aa58-220962241fae" containerName="registry-server"
Jul 14 15:28:06.405544 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:06.405492    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="734be19f-25d9-4b17-aa58-220962241fae" containerName="registry-server"
Jul 14 15:28:06.411856 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:06.411838    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltthc"
Jul 14 15:28:06.414617 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:06.414588    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9ffxp\""
Jul 14 15:28:06.414751 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:06.414694    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 14 15:28:06.418344 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:06.418323    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltthc"]
Jul 14 15:28:06.435345 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:06.435313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t57l\" (UniqueName: \"kubernetes.io/projected/736c5c4a-eda7-4342-8971-ee6c7f0eff1d-kube-api-access-6t57l\") pod \"dns-operator-controller-manager-864dd78c4c-ltthc\" (UID: \"736c5c4a-eda7-4342-8971-ee6c7f0eff1d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltthc"
Jul 14 15:28:06.536493 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:06.536463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t57l\" (UniqueName: \"kubernetes.io/projected/736c5c4a-eda7-4342-8971-ee6c7f0eff1d-kube-api-access-6t57l\") pod \"dns-operator-controller-manager-864dd78c4c-ltthc\" (UID: \"736c5c4a-eda7-4342-8971-ee6c7f0eff1d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltthc"
Jul 14 15:28:06.549634 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:06.549600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t57l\" (UniqueName: \"kubernetes.io/projected/736c5c4a-eda7-4342-8971-ee6c7f0eff1d-kube-api-access-6t57l\") pod \"dns-operator-controller-manager-864dd78c4c-ltthc\" (UID: \"736c5c4a-eda7-4342-8971-ee6c7f0eff1d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltthc"
Jul 14 15:28:06.723839 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:06.723760    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltthc"
Jul 14 15:28:06.861988 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:06.861962    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltthc"]
Jul 14 15:28:06.864439 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:28:06.864410    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736c5c4a_eda7_4342_8971_ee6c7f0eff1d.slice/crio-23c8c2abeaeb8c79cb4bd497f0b8042ac855270f0aea7f7016e4be0aebcc9161 WatchSource:0}: Error finding container 23c8c2abeaeb8c79cb4bd497f0b8042ac855270f0aea7f7016e4be0aebcc9161: Status 404 returned error can't find the container with id 23c8c2abeaeb8c79cb4bd497f0b8042ac855270f0aea7f7016e4be0aebcc9161
Jul 14 15:28:07.206786 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:07.206756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltthc" event={"ID":"736c5c4a-eda7-4342-8971-ee6c7f0eff1d","Type":"ContainerStarted","Data":"23c8c2abeaeb8c79cb4bd497f0b8042ac855270f0aea7f7016e4be0aebcc9161"}
Jul 14 15:28:10.223864 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:10.223820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltthc" event={"ID":"736c5c4a-eda7-4342-8971-ee6c7f0eff1d","Type":"ContainerStarted","Data":"0f08a3c77406007ab0e30e805934092c23919de77171cdf675e77d5969674f72"}
Jul 14 15:28:10.224238 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:10.223886    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltthc"
Jul 14 15:28:10.242488 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:10.242439    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltthc" podStartSLOduration=1.699525615 podStartE2EDuration="4.242423536s" totalImagesPullingTime="2.542897921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:28:10.240695672 +0000 UTC m=+414.581961594" watchObservedRunningTime="2026-07-14 15:28:10.242423536 +0000 UTC m=+414.583689442"
Jul 14 15:28:12.545155 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:12.545118    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"]
Jul 14 15:28:12.548722 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:12.548702    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"
Jul 14 15:28:12.551992 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:12.551968    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-klk96\""
Jul 14 15:28:12.561697 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:12.561677    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"]
Jul 14 15:28:12.596323 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:12.596290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s8w6\" (UniqueName: \"kubernetes.io/projected/28368bb4-ad7d-4a9f-b760-e5c622f2f219-kube-api-access-7s8w6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ph9rq\" (UID: \"28368bb4-ad7d-4a9f-b760-e5c622f2f219\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"
Jul 14 15:28:12.596475 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:12.596358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28368bb4-ad7d-4a9f-b760-e5c622f2f219-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ph9rq\" (UID: \"28368bb4-ad7d-4a9f-b760-e5c622f2f219\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"
Jul 14 15:28:12.697676 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:12.697641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28368bb4-ad7d-4a9f-b760-e5c622f2f219-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ph9rq\" (UID: \"28368bb4-ad7d-4a9f-b760-e5c622f2f219\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"
Jul 14 15:28:12.697854 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:12.697715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s8w6\" (UniqueName: \"kubernetes.io/projected/28368bb4-ad7d-4a9f-b760-e5c622f2f219-kube-api-access-7s8w6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ph9rq\" (UID: \"28368bb4-ad7d-4a9f-b760-e5c622f2f219\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"
Jul 14 15:28:12.698039 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:12.698017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28368bb4-ad7d-4a9f-b760-e5c622f2f219-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ph9rq\" (UID: \"28368bb4-ad7d-4a9f-b760-e5c622f2f219\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"
Jul 14 15:28:12.707119 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:12.707094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s8w6\" (UniqueName: \"kubernetes.io/projected/28368bb4-ad7d-4a9f-b760-e5c622f2f219-kube-api-access-7s8w6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ph9rq\" (UID: \"28368bb4-ad7d-4a9f-b760-e5c622f2f219\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"
Jul 14 15:28:12.858659 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:12.858577    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"
Jul 14 15:28:12.999821 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:12.999779    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"]
Jul 14 15:28:13.002304 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:28:13.002277    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28368bb4_ad7d_4a9f_b760_e5c622f2f219.slice/crio-a1ddb9a95c4a6dbebf7f75cdf308478bbf86c25fc046d5a28d16d7ce2fcc01b0 WatchSource:0}: Error finding container a1ddb9a95c4a6dbebf7f75cdf308478bbf86c25fc046d5a28d16d7ce2fcc01b0: Status 404 returned error can't find the container with id a1ddb9a95c4a6dbebf7f75cdf308478bbf86c25fc046d5a28d16d7ce2fcc01b0
Jul 14 15:28:13.239295 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:13.239259    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq" event={"ID":"28368bb4-ad7d-4a9f-b760-e5c622f2f219","Type":"ContainerStarted","Data":"a1ddb9a95c4a6dbebf7f75cdf308478bbf86c25fc046d5a28d16d7ce2fcc01b0"}
Jul 14 15:28:21.210014 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:21.209980    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kn2cz"]
Jul 14 15:28:21.213700 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:21.213682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kn2cz"
Jul 14 15:28:21.216405 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:21.216385    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-m5drb\""
Jul 14 15:28:21.224884 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:21.224855    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kn2cz"]
Jul 14 15:28:21.231587 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:21.231556    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ltthc"
Jul 14 15:28:21.279035 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:21.278999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4kk7\" (UniqueName: \"kubernetes.io/projected/f7655a49-edcc-4c31-9ecf-41f2ad8dd633-kube-api-access-k4kk7\") pod \"authorino-operator-75cc8d5cb7-kn2cz\" (UID: \"f7655a49-edcc-4c31-9ecf-41f2ad8dd633\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kn2cz"
Jul 14 15:28:21.289014 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:21.288966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq" event={"ID":"28368bb4-ad7d-4a9f-b760-e5c622f2f219","Type":"ContainerStarted","Data":"67d29c161ff7c5b72a07fdddb71ca8ccdb10804e9a283261b31d6f39708b66f9"}
Jul 14 15:28:21.289392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:21.289354    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"
Jul 14 15:28:21.311225 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:21.311177    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq" podStartSLOduration=1.801345103 podStartE2EDuration="9.311161903s" totalImagesPullingTime="7.5098168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:28:21.309648912 +0000 UTC m=+425.650914868" watchObservedRunningTime="2026-07-14 15:28:21.311161903 +0000 UTC m=+425.652427788"
Jul 14 15:28:21.380438 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:21.380391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4kk7\" (UniqueName: \"kubernetes.io/projected/f7655a49-edcc-4c31-9ecf-41f2ad8dd633-kube-api-access-k4kk7\") pod \"authorino-operator-75cc8d5cb7-kn2cz\" (UID: \"f7655a49-edcc-4c31-9ecf-41f2ad8dd633\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kn2cz"
Jul 14 15:28:21.389304 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:21.389277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4kk7\" (UniqueName: \"kubernetes.io/projected/f7655a49-edcc-4c31-9ecf-41f2ad8dd633-kube-api-access-k4kk7\") pod \"authorino-operator-75cc8d5cb7-kn2cz\" (UID: \"f7655a49-edcc-4c31-9ecf-41f2ad8dd633\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kn2cz"
Jul 14 15:28:21.526033 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:21.525942    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kn2cz"
Jul 14 15:28:21.678334 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:21.678310    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kn2cz"]
Jul 14 15:28:21.680410 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:28:21.680378    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7655a49_edcc_4c31_9ecf_41f2ad8dd633.slice/crio-c879240a0e9278a65fb874b542d497930ac02d318e422ed4a80a7ce619db425d WatchSource:0}: Error finding container c879240a0e9278a65fb874b542d497930ac02d318e422ed4a80a7ce619db425d: Status 404 returned error can't find the container with id c879240a0e9278a65fb874b542d497930ac02d318e422ed4a80a7ce619db425d
Jul 14 15:28:22.294564 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:22.294525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kn2cz" event={"ID":"f7655a49-edcc-4c31-9ecf-41f2ad8dd633","Type":"ContainerStarted","Data":"c879240a0e9278a65fb874b542d497930ac02d318e422ed4a80a7ce619db425d"}
Jul 14 15:28:27.323090 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:27.323052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kn2cz" event={"ID":"f7655a49-edcc-4c31-9ecf-41f2ad8dd633","Type":"ContainerStarted","Data":"ac860a0bbf998f1aac95f93a5d11790fbe624af1e9f49e6890662d5e2e9188ef"}
Jul 14 15:28:27.323539 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:27.323210    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kn2cz"
Jul 14 15:28:27.343144 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:27.343101    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kn2cz" podStartSLOduration=1.510827914 podStartE2EDuration="6.343085779s" totalImagesPullingTime="4.832257865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:28:27.341032144 +0000 UTC m=+431.682298049" watchObservedRunningTime="2026-07-14 15:28:27.343085779 +0000 UTC m=+431.684351686"
Jul 14 15:28:32.296554 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:32.296490    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"
Jul 14 15:28:33.591114 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:33.591080    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"]
Jul 14 15:28:33.594951 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:33.594933    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:33.608694 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:33.608670    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"]
Jul 14 15:28:33.701891 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:33.701862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77524685-19aa-4d61-9c71-51d5a159f372-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-98gh2\" (UID: \"77524685-19aa-4d61-9c71-51d5a159f372\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:33.702094 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:33.701926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bthxp\" (UniqueName: \"kubernetes.io/projected/77524685-19aa-4d61-9c71-51d5a159f372-kube-api-access-bthxp\") pod \"kuadrant-operator-controller-manager-55467d9f7-98gh2\" (UID: \"77524685-19aa-4d61-9c71-51d5a159f372\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:33.802604 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:33.802565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bthxp\" (UniqueName: \"kubernetes.io/projected/77524685-19aa-4d61-9c71-51d5a159f372-kube-api-access-bthxp\") pod \"kuadrant-operator-controller-manager-55467d9f7-98gh2\" (UID: \"77524685-19aa-4d61-9c71-51d5a159f372\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:33.802801 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:33.802672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77524685-19aa-4d61-9c71-51d5a159f372-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-98gh2\" (UID: \"77524685-19aa-4d61-9c71-51d5a159f372\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:33.803046 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:33.803028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77524685-19aa-4d61-9c71-51d5a159f372-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-98gh2\" (UID: \"77524685-19aa-4d61-9c71-51d5a159f372\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:33.814578 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:33.814550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bthxp\" (UniqueName: \"kubernetes.io/projected/77524685-19aa-4d61-9c71-51d5a159f372-kube-api-access-bthxp\") pod \"kuadrant-operator-controller-manager-55467d9f7-98gh2\" (UID: \"77524685-19aa-4d61-9c71-51d5a159f372\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:33.905674 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:33.905643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:34.044680 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.044659    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"]
Jul 14 15:28:34.046393 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:28:34.046368    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77524685_19aa_4d61_9c71_51d5a159f372.slice/crio-7f55f3418255cdafa070488a2071097aadc720e1c861dd10a49c5615d59abaaf WatchSource:0}: Error finding container 7f55f3418255cdafa070488a2071097aadc720e1c861dd10a49c5615d59abaaf: Status 404 returned error can't find the container with id 7f55f3418255cdafa070488a2071097aadc720e1c861dd10a49c5615d59abaaf
Jul 14 15:28:34.160539 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.160442    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"]
Jul 14 15:28:34.164834 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.164804    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"]
Jul 14 15:28:34.176526 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.176480    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"]
Jul 14 15:28:34.176789 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.176730    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq" podUID="28368bb4-ad7d-4a9f-b760-e5c622f2f219" containerName="manager" containerID="cri-o://67d29c161ff7c5b72a07fdddb71ca8ccdb10804e9a283261b31d6f39708b66f9" gracePeriod=2
Jul 14 15:28:34.187436 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.187405    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"]
Jul 14 15:28:34.197695 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.197669    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"]
Jul 14 15:28:34.198055 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.198044    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28368bb4-ad7d-4a9f-b760-e5c622f2f219" containerName="manager"
Jul 14 15:28:34.198095 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.198058    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="28368bb4-ad7d-4a9f-b760-e5c622f2f219" containerName="manager"
Jul 14 15:28:34.198095 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.198072    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77524685-19aa-4d61-9c71-51d5a159f372" containerName="manager"
Jul 14 15:28:34.198095 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.198079    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="77524685-19aa-4d61-9c71-51d5a159f372" containerName="manager"
Jul 14 15:28:34.198187 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.198141    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="77524685-19aa-4d61-9c71-51d5a159f372" containerName="manager"
Jul 14 15:28:34.198187 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.198151    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="28368bb4-ad7d-4a9f-b760-e5c622f2f219" containerName="manager"
Jul 14 15:28:34.201086 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.201071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"
Jul 14 15:28:34.217954 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.217927    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"]
Jul 14 15:28:34.224735 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.224705    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z2pt5"]
Jul 14 15:28:34.227427 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:28:34.227395    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ph9rq\" is forbidden: User \"system:node:ip-10-0-136-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-210.ec2.internal' and this object" podUID="28368bb4-ad7d-4a9f-b760-e5c622f2f219" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"
Jul 14 15:28:34.229092 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.229071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z2pt5"
Jul 14 15:28:34.233902 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.233667    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-j9tbp\""
Jul 14 15:28:34.236101 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.236082    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z2pt5"]
Jul 14 15:28:34.314610 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.314582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q9gs\" (UniqueName: \"kubernetes.io/projected/8d763c04-1a1e-409a-9ced-aad811881e54-kube-api-access-9q9gs\") pod \"limitador-operator-controller-manager-859d7b78bd-z2pt5\" (UID: \"8d763c04-1a1e-409a-9ced-aad811881e54\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z2pt5"
Jul 14 15:28:34.314761 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.314630    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9h9g\" (UniqueName: \"kubernetes.io/projected/abf84985-4a85-4d60-8ec0-872ca9af256b-kube-api-access-x9h9g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rk8hf\" (UID: \"abf84985-4a85-4d60-8ec0-872ca9af256b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"
Jul 14 15:28:34.314761 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.314692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abf84985-4a85-4d60-8ec0-872ca9af256b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rk8hf\" (UID: \"abf84985-4a85-4d60-8ec0-872ca9af256b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"
Jul 14 15:28:34.359671 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.359646    2563 generic.go:356] "Generic (PLEG): container finished" podID="28368bb4-ad7d-4a9f-b760-e5c622f2f219" containerID="67d29c161ff7c5b72a07fdddb71ca8ccdb10804e9a283261b31d6f39708b66f9" exitCode=0
Jul 14 15:28:34.361490 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:28:34.361474    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2" container="manager"
Jul 14 15:28:34.409942 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.409917    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"
Jul 14 15:28:34.415137 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.415088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9h9g\" (UniqueName: \"kubernetes.io/projected/abf84985-4a85-4d60-8ec0-872ca9af256b-kube-api-access-x9h9g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rk8hf\" (UID: \"abf84985-4a85-4d60-8ec0-872ca9af256b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"
Jul 14 15:28:34.415253 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.415132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abf84985-4a85-4d60-8ec0-872ca9af256b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rk8hf\" (UID: \"abf84985-4a85-4d60-8ec0-872ca9af256b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"
Jul 14 15:28:34.415310 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.415278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q9gs\" (UniqueName: \"kubernetes.io/projected/8d763c04-1a1e-409a-9ced-aad811881e54-kube-api-access-9q9gs\") pod \"limitador-operator-controller-manager-859d7b78bd-z2pt5\" (UID: \"8d763c04-1a1e-409a-9ced-aad811881e54\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z2pt5"
Jul 14 15:28:34.415486 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.415470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abf84985-4a85-4d60-8ec0-872ca9af256b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rk8hf\" (UID: \"abf84985-4a85-4d60-8ec0-872ca9af256b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"
Jul 14 15:28:34.428962 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.428932    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q9gs\" (UniqueName: \"kubernetes.io/projected/8d763c04-1a1e-409a-9ced-aad811881e54-kube-api-access-9q9gs\") pod \"limitador-operator-controller-manager-859d7b78bd-z2pt5\" (UID: \"8d763c04-1a1e-409a-9ced-aad811881e54\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z2pt5"
Jul 14 15:28:34.428962 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.428958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9h9g\" (UniqueName: \"kubernetes.io/projected/abf84985-4a85-4d60-8ec0-872ca9af256b-kube-api-access-x9h9g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rk8hf\" (UID: \"abf84985-4a85-4d60-8ec0-872ca9af256b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"
Jul 14 15:28:34.515931 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.515883    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s8w6\" (UniqueName: \"kubernetes.io/projected/28368bb4-ad7d-4a9f-b760-e5c622f2f219-kube-api-access-7s8w6\") pod \"28368bb4-ad7d-4a9f-b760-e5c622f2f219\" (UID: \"28368bb4-ad7d-4a9f-b760-e5c622f2f219\") "
Jul 14 15:28:34.516100 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.515958    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28368bb4-ad7d-4a9f-b760-e5c622f2f219-extensions-socket-volume\") pod \"28368bb4-ad7d-4a9f-b760-e5c622f2f219\" (UID: \"28368bb4-ad7d-4a9f-b760-e5c622f2f219\") "
Jul 14 15:28:34.516406 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.516383    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28368bb4-ad7d-4a9f-b760-e5c622f2f219-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "28368bb4-ad7d-4a9f-b760-e5c622f2f219" (UID: "28368bb4-ad7d-4a9f-b760-e5c622f2f219"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:28:34.518114 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.518093    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28368bb4-ad7d-4a9f-b760-e5c622f2f219-kube-api-access-7s8w6" (OuterVolumeSpecName: "kube-api-access-7s8w6") pod "28368bb4-ad7d-4a9f-b760-e5c622f2f219" (UID: "28368bb4-ad7d-4a9f-b760-e5c622f2f219"). InnerVolumeSpecName "kube-api-access-7s8w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:28:34.544842 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.544814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"
Jul 14 15:28:34.550948 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.550931    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z2pt5"
Jul 14 15:28:34.617882 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.617849    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s8w6\" (UniqueName: \"kubernetes.io/projected/28368bb4-ad7d-4a9f-b760-e5c622f2f219-kube-api-access-7s8w6\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:28:34.618261 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.617884    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28368bb4-ad7d-4a9f-b760-e5c622f2f219-extensions-socket-volume\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:28:34.686808 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.686782    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"]
Jul 14 15:28:34.689579 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:28:34.689554    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf84985_4a85_4d60_8ec0_872ca9af256b.slice/crio-044b3d62406f9020a219d588a18c461b94f0795d0a9f19b0b97327bba9b80b80 WatchSource:0}: Error finding container 044b3d62406f9020a219d588a18c461b94f0795d0a9f19b0b97327bba9b80b80: Status 404 returned error can't find the container with id 044b3d62406f9020a219d588a18c461b94f0795d0a9f19b0b97327bba9b80b80
Jul 14 15:28:34.708619 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:34.708598    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z2pt5"]
Jul 14 15:28:34.710860 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:28:34.710837    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d763c04_1a1e_409a_9ced_aad811881e54.slice/crio-2322e3594bcf6044300c7ba1ea84635583e3ca27b6ea887252e96a0ed459f800 WatchSource:0}: Error finding container 2322e3594bcf6044300c7ba1ea84635583e3ca27b6ea887252e96a0ed459f800: Status 404 returned error can't find the container with id 2322e3594bcf6044300c7ba1ea84635583e3ca27b6ea887252e96a0ed459f800
Jul 14 15:28:35.370820 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.370688    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2" podUID="77524685-19aa-4d61-9c71-51d5a159f372" containerName="manager" containerID="cri-o://8062b5a83814397f76ab61826043518abca25cdd90a9048d47147c6d9b616af3" gracePeriod=2
Jul 14 15:28:35.372772 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.372740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z2pt5" event={"ID":"8d763c04-1a1e-409a-9ced-aad811881e54","Type":"ContainerStarted","Data":"2322e3594bcf6044300c7ba1ea84635583e3ca27b6ea887252e96a0ed459f800"}
Jul 14 15:28:35.373287 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:28:35.373254    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-98gh2\" is forbidden: User \"system:node:ip-10-0-136-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-210.ec2.internal' and this object" podUID="77524685-19aa-4d61-9c71-51d5a159f372" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:35.374807 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.374782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf" event={"ID":"abf84985-4a85-4d60-8ec0-872ca9af256b","Type":"ContainerStarted","Data":"b4cd41dec8d81fcb939e6a3926cc1e4b1c23e7b008018f3c8e2e4d1e9a82476e"}
Jul 14 15:28:35.374930 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.374834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf" event={"ID":"abf84985-4a85-4d60-8ec0-872ca9af256b","Type":"ContainerStarted","Data":"044b3d62406f9020a219d588a18c461b94f0795d0a9f19b0b97327bba9b80b80"}
Jul 14 15:28:35.375193 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.375114    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"
Jul 14 15:28:35.377256 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.377117    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ph9rq"
Jul 14 15:28:35.377256 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.377138    2563 scope.go:117] "RemoveContainer" containerID="67d29c161ff7c5b72a07fdddb71ca8ccdb10804e9a283261b31d6f39708b66f9"
Jul 14 15:28:35.397073 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:28:35.397036    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-98gh2\" is forbidden: User \"system:node:ip-10-0-136-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-210.ec2.internal' and this object" podUID="77524685-19aa-4d61-9c71-51d5a159f372" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:35.398692 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.398641    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf" podStartSLOduration=1.398621823 podStartE2EDuration="1.398621823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:28:35.394395976 +0000 UTC m=+439.735661885" watchObservedRunningTime="2026-07-14 15:28:35.398621823 +0000 UTC m=+439.739887731"
Jul 14 15:28:35.399610 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:28:35.399580    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-98gh2\" is forbidden: User \"system:node:ip-10-0-136-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-210.ec2.internal' and this object" podUID="77524685-19aa-4d61-9c71-51d5a159f372" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:35.426724 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:28:35.426678    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-98gh2\" is forbidden: User \"system:node:ip-10-0-136-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-210.ec2.internal' and this object" podUID="77524685-19aa-4d61-9c71-51d5a159f372" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:35.643487 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.643459    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:35.646493 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:28:35.646459    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-98gh2\" is forbidden: User \"system:node:ip-10-0-136-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-210.ec2.internal' and this object" podUID="77524685-19aa-4d61-9c71-51d5a159f372" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:35.728741 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.728701    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bthxp\" (UniqueName: \"kubernetes.io/projected/77524685-19aa-4d61-9c71-51d5a159f372-kube-api-access-bthxp\") pod \"77524685-19aa-4d61-9c71-51d5a159f372\" (UID: \"77524685-19aa-4d61-9c71-51d5a159f372\") "
Jul 14 15:28:35.728911 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.728764    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77524685-19aa-4d61-9c71-51d5a159f372-extensions-socket-volume\") pod \"77524685-19aa-4d61-9c71-51d5a159f372\" (UID: \"77524685-19aa-4d61-9c71-51d5a159f372\") "
Jul 14 15:28:35.728999 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.728968    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77524685-19aa-4d61-9c71-51d5a159f372-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "77524685-19aa-4d61-9c71-51d5a159f372" (UID: "77524685-19aa-4d61-9c71-51d5a159f372"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:28:35.729083 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.729069    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77524685-19aa-4d61-9c71-51d5a159f372-extensions-socket-volume\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:28:35.730869 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.730849    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77524685-19aa-4d61-9c71-51d5a159f372-kube-api-access-bthxp" (OuterVolumeSpecName: "kube-api-access-bthxp") pod "77524685-19aa-4d61-9c71-51d5a159f372" (UID: "77524685-19aa-4d61-9c71-51d5a159f372"). InnerVolumeSpecName "kube-api-access-bthxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:28:35.830462 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:35.830429    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bthxp\" (UniqueName: \"kubernetes.io/projected/77524685-19aa-4d61-9c71-51d5a159f372-kube-api-access-bthxp\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:28:36.215303 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:28:36.215268    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-98gh2\" is forbidden: User \"system:node:ip-10-0-136-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-210.ec2.internal' and this object" podUID="77524685-19aa-4d61-9c71-51d5a159f372" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:36.216892 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:36.216870    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28368bb4-ad7d-4a9f-b760-e5c622f2f219" path="/var/lib/kubelet/pods/28368bb4-ad7d-4a9f-b760-e5c622f2f219/volumes"
Jul 14 15:28:36.217283 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:36.217262    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77524685-19aa-4d61-9c71-51d5a159f372" path="/var/lib/kubelet/pods/77524685-19aa-4d61-9c71-51d5a159f372/volumes"
Jul 14 15:28:36.383720 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:36.383698    2563 generic.go:356] "Generic (PLEG): container finished" podID="77524685-19aa-4d61-9c71-51d5a159f372" containerID="8062b5a83814397f76ab61826043518abca25cdd90a9048d47147c6d9b616af3" exitCode=2
Jul 14 15:28:36.383816 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:36.383745    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:36.383816 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:36.383801    2563 scope.go:117] "RemoveContainer" containerID="8062b5a83814397f76ab61826043518abca25cdd90a9048d47147c6d9b616af3"
Jul 14 15:28:36.389198 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:28:36.389173    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-98gh2\" is forbidden: User \"system:node:ip-10-0-136-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-210.ec2.internal' and this object" podUID="77524685-19aa-4d61-9c71-51d5a159f372" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:36.394302 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:36.394279    2563 scope.go:117] "RemoveContainer" containerID="8062b5a83814397f76ab61826043518abca25cdd90a9048d47147c6d9b616af3"
Jul 14 15:28:36.394652 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:28:36.394629    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8062b5a83814397f76ab61826043518abca25cdd90a9048d47147c6d9b616af3\": container with ID starting with 8062b5a83814397f76ab61826043518abca25cdd90a9048d47147c6d9b616af3 not found: ID does not exist" containerID="8062b5a83814397f76ab61826043518abca25cdd90a9048d47147c6d9b616af3"
Jul 14 15:28:36.394734 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:36.394659    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8062b5a83814397f76ab61826043518abca25cdd90a9048d47147c6d9b616af3"} err="failed to get container status \"8062b5a83814397f76ab61826043518abca25cdd90a9048d47147c6d9b616af3\": rpc error: code = NotFound desc = could not find container \"8062b5a83814397f76ab61826043518abca25cdd90a9048d47147c6d9b616af3\": container with ID starting with 8062b5a83814397f76ab61826043518abca25cdd90a9048d47147c6d9b616af3 not found: ID does not exist"
Jul 14 15:28:37.393274 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:37.393236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z2pt5" event={"ID":"8d763c04-1a1e-409a-9ced-aad811881e54","Type":"ContainerStarted","Data":"f30e70161c14b8a10e874081dc7b9de9f746a8c9e7c4cd7110ec35e6b98debce"}
Jul 14 15:28:37.393808 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:37.393349    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z2pt5"
Jul 14 15:28:37.395785 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:28:37.395755    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-98gh2\" is forbidden: User \"system:node:ip-10-0-136-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-210.ec2.internal' and this object" podUID="77524685-19aa-4d61-9c71-51d5a159f372" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-98gh2"
Jul 14 15:28:37.413789 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:37.413738    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z2pt5" podStartSLOduration=1.747585153 podStartE2EDuration="3.413721196s" totalImagesPullingTime="1.666136043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:28:37.411885 +0000 UTC m=+441.753150935" watchObservedRunningTime="2026-07-14 15:28:37.413721196 +0000 UTC m=+441.754987102"
Jul 14 15:28:38.329563 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:38.329536    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kn2cz"
Jul 14 15:28:46.387839 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:46.387810    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"
Jul 14 15:28:48.400141 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:48.400111    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z2pt5"
Jul 14 15:28:56.975597 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:56.975567    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75688698d8-jtfr2"]
Jul 14 15:28:56.979253 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:56.979231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:56.990722 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:56.990698    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75688698d8-jtfr2"]
Jul 14 15:28:57.002703 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.002673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d060879f-e91e-40bd-93ea-11c416eaad8a-service-ca\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.002823 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.002718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d060879f-e91e-40bd-93ea-11c416eaad8a-console-config\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.002823 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.002742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d060879f-e91e-40bd-93ea-11c416eaad8a-oauth-serving-cert\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.002823 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.002765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d060879f-e91e-40bd-93ea-11c416eaad8a-console-serving-cert\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.002823 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.002811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d060879f-e91e-40bd-93ea-11c416eaad8a-trusted-ca-bundle\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.002956 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.002846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d060879f-e91e-40bd-93ea-11c416eaad8a-console-oauth-config\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.002956 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.002885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49hn\" (UniqueName: \"kubernetes.io/projected/d060879f-e91e-40bd-93ea-11c416eaad8a-kube-api-access-d49hn\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.103723 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.103690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d060879f-e91e-40bd-93ea-11c416eaad8a-trusted-ca-bundle\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.103723 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.103733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d060879f-e91e-40bd-93ea-11c416eaad8a-console-oauth-config\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.103971 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.103779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49hn\" (UniqueName: \"kubernetes.io/projected/d060879f-e91e-40bd-93ea-11c416eaad8a-kube-api-access-d49hn\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.103971 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.103839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d060879f-e91e-40bd-93ea-11c416eaad8a-service-ca\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.103971 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.103894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d060879f-e91e-40bd-93ea-11c416eaad8a-console-config\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.103971 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.103917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d060879f-e91e-40bd-93ea-11c416eaad8a-oauth-serving-cert\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.103971 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.103947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d060879f-e91e-40bd-93ea-11c416eaad8a-console-serving-cert\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.104630 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.104600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d060879f-e91e-40bd-93ea-11c416eaad8a-service-ca\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.104757 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.104644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d060879f-e91e-40bd-93ea-11c416eaad8a-console-config\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.104757 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.104670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d060879f-e91e-40bd-93ea-11c416eaad8a-oauth-serving-cert\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.104757 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.104674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d060879f-e91e-40bd-93ea-11c416eaad8a-trusted-ca-bundle\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.106655 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.106638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d060879f-e91e-40bd-93ea-11c416eaad8a-console-serving-cert\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.106713 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.106662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d060879f-e91e-40bd-93ea-11c416eaad8a-console-oauth-config\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.112697 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.112670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49hn\" (UniqueName: \"kubernetes.io/projected/d060879f-e91e-40bd-93ea-11c416eaad8a-kube-api-access-d49hn\") pod \"console-75688698d8-jtfr2\" (UID: \"d060879f-e91e-40bd-93ea-11c416eaad8a\") " pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.288001 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.287907    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:28:57.431759 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.431717    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75688698d8-jtfr2"]
Jul 14 15:28:57.433312 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:28:57.433282    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd060879f_e91e_40bd_93ea_11c416eaad8a.slice/crio-e0cadc0fa72c97ebad8fbb7b09656e0085c3408651f2bda2a044c7e68bca04c6 WatchSource:0}: Error finding container e0cadc0fa72c97ebad8fbb7b09656e0085c3408651f2bda2a044c7e68bca04c6: Status 404 returned error can't find the container with id e0cadc0fa72c97ebad8fbb7b09656e0085c3408651f2bda2a044c7e68bca04c6
Jul 14 15:28:57.492490 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.492460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75688698d8-jtfr2" event={"ID":"d060879f-e91e-40bd-93ea-11c416eaad8a","Type":"ContainerStarted","Data":"77e704215c7a68dfe91f20763703dd69ae615f5684dccbe0e64c6e77dc9f9f60"}
Jul 14 15:28:57.492619 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.492554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75688698d8-jtfr2" event={"ID":"d060879f-e91e-40bd-93ea-11c416eaad8a","Type":"ContainerStarted","Data":"e0cadc0fa72c97ebad8fbb7b09656e0085c3408651f2bda2a044c7e68bca04c6"}
Jul 14 15:28:57.511642 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:28:57.511593    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75688698d8-jtfr2" podStartSLOduration=1.511578452 podStartE2EDuration="1.511578452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:28:57.509924362 +0000 UTC m=+461.851190287" watchObservedRunningTime="2026-07-14 15:28:57.511578452 +0000 UTC m=+461.852844360"
Jul 14 15:29:04.721237 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:04.721202    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"]
Jul 14 15:29:04.721689 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:04.721441    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf" podUID="abf84985-4a85-4d60-8ec0-872ca9af256b" containerName="manager" containerID="cri-o://b4cd41dec8d81fcb939e6a3926cc1e4b1c23e7b008018f3c8e2e4d1e9a82476e" gracePeriod=10
Jul 14 15:29:04.970586 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:04.970556    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"]
Jul 14 15:29:04.975177 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:04.975114    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"
Jul 14 15:29:04.981737 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:04.981712    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"
Jul 14 15:29:04.987677 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:04.987652    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"]
Jul 14 15:29:05.074323 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.074285    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abf84985-4a85-4d60-8ec0-872ca9af256b-extensions-socket-volume\") pod \"abf84985-4a85-4d60-8ec0-872ca9af256b\" (UID: \"abf84985-4a85-4d60-8ec0-872ca9af256b\") "
Jul 14 15:29:05.074494 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.074413    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9h9g\" (UniqueName: \"kubernetes.io/projected/abf84985-4a85-4d60-8ec0-872ca9af256b-kube-api-access-x9h9g\") pod \"abf84985-4a85-4d60-8ec0-872ca9af256b\" (UID: \"abf84985-4a85-4d60-8ec0-872ca9af256b\") "
Jul 14 15:29:05.074572 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.074553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e07be5de-d55a-46dd-b0a8-28674fe9ef6a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mlfxv\" (UID: \"e07be5de-d55a-46dd-b0a8-28674fe9ef6a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"
Jul 14 15:29:05.074663 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.074651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrhnw\" (UniqueName: \"kubernetes.io/projected/e07be5de-d55a-46dd-b0a8-28674fe9ef6a-kube-api-access-xrhnw\") pod \"kuadrant-operator-controller-manager-86878c769f-mlfxv\" (UID: \"e07be5de-d55a-46dd-b0a8-28674fe9ef6a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"
Jul 14 15:29:05.074730 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.074703    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf84985-4a85-4d60-8ec0-872ca9af256b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "abf84985-4a85-4d60-8ec0-872ca9af256b" (UID: "abf84985-4a85-4d60-8ec0-872ca9af256b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:29:05.076646 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.076622    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf84985-4a85-4d60-8ec0-872ca9af256b-kube-api-access-x9h9g" (OuterVolumeSpecName: "kube-api-access-x9h9g") pod "abf84985-4a85-4d60-8ec0-872ca9af256b" (UID: "abf84985-4a85-4d60-8ec0-872ca9af256b"). InnerVolumeSpecName "kube-api-access-x9h9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:29:05.175551 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.175471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrhnw\" (UniqueName: \"kubernetes.io/projected/e07be5de-d55a-46dd-b0a8-28674fe9ef6a-kube-api-access-xrhnw\") pod \"kuadrant-operator-controller-manager-86878c769f-mlfxv\" (UID: \"e07be5de-d55a-46dd-b0a8-28674fe9ef6a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"
Jul 14 15:29:05.175728 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.175569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e07be5de-d55a-46dd-b0a8-28674fe9ef6a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mlfxv\" (UID: \"e07be5de-d55a-46dd-b0a8-28674fe9ef6a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"
Jul 14 15:29:05.175728 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.175662    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9h9g\" (UniqueName: \"kubernetes.io/projected/abf84985-4a85-4d60-8ec0-872ca9af256b-kube-api-access-x9h9g\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:29:05.175728 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.175673    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abf84985-4a85-4d60-8ec0-872ca9af256b-extensions-socket-volume\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:29:05.176016 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.175992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e07be5de-d55a-46dd-b0a8-28674fe9ef6a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mlfxv\" (UID: \"e07be5de-d55a-46dd-b0a8-28674fe9ef6a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"
Jul 14 15:29:05.185376 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.185341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrhnw\" (UniqueName: \"kubernetes.io/projected/e07be5de-d55a-46dd-b0a8-28674fe9ef6a-kube-api-access-xrhnw\") pod \"kuadrant-operator-controller-manager-86878c769f-mlfxv\" (UID: \"e07be5de-d55a-46dd-b0a8-28674fe9ef6a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"
Jul 14 15:29:05.288666 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.288577    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"
Jul 14 15:29:05.438743 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.438721    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"]
Jul 14 15:29:05.440983 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:29:05.440953    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07be5de_d55a_46dd_b0a8_28674fe9ef6a.slice/crio-c700b506f9340ebfd1686f6a3da1a5cedc9208b8f30872bc4dc50931b17a26ec WatchSource:0}: Error finding container c700b506f9340ebfd1686f6a3da1a5cedc9208b8f30872bc4dc50931b17a26ec: Status 404 returned error can't find the container with id c700b506f9340ebfd1686f6a3da1a5cedc9208b8f30872bc4dc50931b17a26ec
Jul 14 15:29:05.535968 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.535934    2563 generic.go:356] "Generic (PLEG): container finished" podID="abf84985-4a85-4d60-8ec0-872ca9af256b" containerID="b4cd41dec8d81fcb939e6a3926cc1e4b1c23e7b008018f3c8e2e4d1e9a82476e" exitCode=0
Jul 14 15:29:05.536196 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.536002    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"
Jul 14 15:29:05.536196 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.536023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf" event={"ID":"abf84985-4a85-4d60-8ec0-872ca9af256b","Type":"ContainerDied","Data":"b4cd41dec8d81fcb939e6a3926cc1e4b1c23e7b008018f3c8e2e4d1e9a82476e"}
Jul 14 15:29:05.536196 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.536074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf" event={"ID":"abf84985-4a85-4d60-8ec0-872ca9af256b","Type":"ContainerDied","Data":"044b3d62406f9020a219d588a18c461b94f0795d0a9f19b0b97327bba9b80b80"}
Jul 14 15:29:05.536196 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.536103    2563 scope.go:117] "RemoveContainer" containerID="b4cd41dec8d81fcb939e6a3926cc1e4b1c23e7b008018f3c8e2e4d1e9a82476e"
Jul 14 15:29:05.537932 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.537904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv" event={"ID":"e07be5de-d55a-46dd-b0a8-28674fe9ef6a","Type":"ContainerStarted","Data":"477a4482be4a1b49568f2fb92bff241772ebac77e240dbdb4fb56451b2ab9538"}
Jul 14 15:29:05.538051 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.537953    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv" event={"ID":"e07be5de-d55a-46dd-b0a8-28674fe9ef6a","Type":"ContainerStarted","Data":"c700b506f9340ebfd1686f6a3da1a5cedc9208b8f30872bc4dc50931b17a26ec"}
Jul 14 15:29:05.538105 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.538057    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"
Jul 14 15:29:05.545679 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.545658    2563 scope.go:117] "RemoveContainer" containerID="b4cd41dec8d81fcb939e6a3926cc1e4b1c23e7b008018f3c8e2e4d1e9a82476e"
Jul 14 15:29:05.545953 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:29:05.545933    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4cd41dec8d81fcb939e6a3926cc1e4b1c23e7b008018f3c8e2e4d1e9a82476e\": container with ID starting with b4cd41dec8d81fcb939e6a3926cc1e4b1c23e7b008018f3c8e2e4d1e9a82476e not found: ID does not exist" containerID="b4cd41dec8d81fcb939e6a3926cc1e4b1c23e7b008018f3c8e2e4d1e9a82476e"
Jul 14 15:29:05.546026 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.545961    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4cd41dec8d81fcb939e6a3926cc1e4b1c23e7b008018f3c8e2e4d1e9a82476e"} err="failed to get container status \"b4cd41dec8d81fcb939e6a3926cc1e4b1c23e7b008018f3c8e2e4d1e9a82476e\": rpc error: code = NotFound desc = could not find container \"b4cd41dec8d81fcb939e6a3926cc1e4b1c23e7b008018f3c8e2e4d1e9a82476e\": container with ID starting with b4cd41dec8d81fcb939e6a3926cc1e4b1c23e7b008018f3c8e2e4d1e9a82476e not found: ID does not exist"
Jul 14 15:29:05.561088 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.561040    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv" podStartSLOduration=1.561027902 podStartE2EDuration="1.561027902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:29:05.559481708 +0000 UTC m=+469.900747632" watchObservedRunningTime="2026-07-14 15:29:05.561027902 +0000 UTC m=+469.902293881"
Jul 14 15:29:05.577364 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.577336    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"]
Jul 14 15:29:05.584179 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:05.584156    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rk8hf"]
Jul 14 15:29:06.216619 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:06.216588    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abf84985-4a85-4d60-8ec0-872ca9af256b" path="/var/lib/kubelet/pods/abf84985-4a85-4d60-8ec0-872ca9af256b/volumes"
Jul 14 15:29:07.288283 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:07.288252    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:29:07.288762 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:07.288326    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:29:07.293324 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:07.293291    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:29:07.552625 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:07.552549    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75688698d8-jtfr2"
Jul 14 15:29:07.611387 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:07.611357    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f7c8ff86-dbtwv"]
Jul 14 15:29:16.546341 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:16.546313    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"
Jul 14 15:29:32.633780 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.633742    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67f7c8ff86-dbtwv" podUID="88a4571e-bc89-4552-99d9-ac3747bbbec8" containerName="console" containerID="cri-o://558ef529829f6719ce73b97c365a08fb4e33709857c8949c7dd42b0104dfd3dc" gracePeriod=15
Jul 14 15:29:32.904714 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.904687    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:29:32.951181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.951147    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-oauth-serving-cert\") pod \"88a4571e-bc89-4552-99d9-ac3747bbbec8\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") "
Jul 14 15:29:32.951358 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.951195    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-trusted-ca-bundle\") pod \"88a4571e-bc89-4552-99d9-ac3747bbbec8\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") "
Jul 14 15:29:32.951358 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.951223    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-config\") pod \"88a4571e-bc89-4552-99d9-ac3747bbbec8\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") "
Jul 14 15:29:32.951358 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.951301    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-serving-cert\") pod \"88a4571e-bc89-4552-99d9-ac3747bbbec8\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") "
Jul 14 15:29:32.951358 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.951327    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cxlk\" (UniqueName: \"kubernetes.io/projected/88a4571e-bc89-4552-99d9-ac3747bbbec8-kube-api-access-9cxlk\") pod \"88a4571e-bc89-4552-99d9-ac3747bbbec8\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") "
Jul 14 15:29:32.951595 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.951361    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-service-ca\") pod \"88a4571e-bc89-4552-99d9-ac3747bbbec8\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") "
Jul 14 15:29:32.951595 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.951386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-oauth-config\") pod \"88a4571e-bc89-4552-99d9-ac3747bbbec8\" (UID: \"88a4571e-bc89-4552-99d9-ac3747bbbec8\") "
Jul 14 15:29:32.951694 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.951614    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "88a4571e-bc89-4552-99d9-ac3747bbbec8" (UID: "88a4571e-bc89-4552-99d9-ac3747bbbec8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:29:32.951744 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.951718    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-oauth-serving-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:29:32.951787 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.951759    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "88a4571e-bc89-4552-99d9-ac3747bbbec8" (UID: "88a4571e-bc89-4552-99d9-ac3747bbbec8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:29:32.951895 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.951863    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-service-ca" (OuterVolumeSpecName: "service-ca") pod "88a4571e-bc89-4552-99d9-ac3747bbbec8" (UID: "88a4571e-bc89-4552-99d9-ac3747bbbec8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:29:32.952009 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.951993    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-config" (OuterVolumeSpecName: "console-config") pod "88a4571e-bc89-4552-99d9-ac3747bbbec8" (UID: "88a4571e-bc89-4552-99d9-ac3747bbbec8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:29:32.953603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.953579    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a4571e-bc89-4552-99d9-ac3747bbbec8-kube-api-access-9cxlk" (OuterVolumeSpecName: "kube-api-access-9cxlk") pod "88a4571e-bc89-4552-99d9-ac3747bbbec8" (UID: "88a4571e-bc89-4552-99d9-ac3747bbbec8"). InnerVolumeSpecName "kube-api-access-9cxlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:29:32.953718 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.953699    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "88a4571e-bc89-4552-99d9-ac3747bbbec8" (UID: "88a4571e-bc89-4552-99d9-ac3747bbbec8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:29:32.953718 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:32.953712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "88a4571e-bc89-4552-99d9-ac3747bbbec8" (UID: "88a4571e-bc89-4552-99d9-ac3747bbbec8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:29:33.052181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.052145    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-serving-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:29:33.052181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.052175    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cxlk\" (UniqueName: \"kubernetes.io/projected/88a4571e-bc89-4552-99d9-ac3747bbbec8-kube-api-access-9cxlk\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:29:33.052181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.052186    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:29:33.052409 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.052196    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-oauth-config\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:29:33.052409 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.052205    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-trusted-ca-bundle\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:29:33.052409 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.052214    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a4571e-bc89-4552-99d9-ac3747bbbec8-console-config\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:29:33.680688 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.680653    2563 generic.go:356] "Generic (PLEG): container finished" podID="88a4571e-bc89-4552-99d9-ac3747bbbec8" containerID="558ef529829f6719ce73b97c365a08fb4e33709857c8949c7dd42b0104dfd3dc" exitCode=0
Jul 14 15:29:33.681131 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.680700    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7c8ff86-dbtwv"
Jul 14 15:29:33.681131 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.680735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7c8ff86-dbtwv" event={"ID":"88a4571e-bc89-4552-99d9-ac3747bbbec8","Type":"ContainerDied","Data":"558ef529829f6719ce73b97c365a08fb4e33709857c8949c7dd42b0104dfd3dc"}
Jul 14 15:29:33.681131 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.680786    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7c8ff86-dbtwv" event={"ID":"88a4571e-bc89-4552-99d9-ac3747bbbec8","Type":"ContainerDied","Data":"24306f966672604c8e2452562b4ac5b37d720560a69941d0ee9b054b712b5f89"}
Jul 14 15:29:33.681131 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.680807    2563 scope.go:117] "RemoveContainer" containerID="558ef529829f6719ce73b97c365a08fb4e33709857c8949c7dd42b0104dfd3dc"
Jul 14 15:29:33.690542 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.690494    2563 scope.go:117] "RemoveContainer" containerID="558ef529829f6719ce73b97c365a08fb4e33709857c8949c7dd42b0104dfd3dc"
Jul 14 15:29:33.690797 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:29:33.690780    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558ef529829f6719ce73b97c365a08fb4e33709857c8949c7dd42b0104dfd3dc\": container with ID starting with 558ef529829f6719ce73b97c365a08fb4e33709857c8949c7dd42b0104dfd3dc not found: ID does not exist" containerID="558ef529829f6719ce73b97c365a08fb4e33709857c8949c7dd42b0104dfd3dc"
Jul 14 15:29:33.690875 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.690822    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558ef529829f6719ce73b97c365a08fb4e33709857c8949c7dd42b0104dfd3dc"} err="failed to get container status \"558ef529829f6719ce73b97c365a08fb4e33709857c8949c7dd42b0104dfd3dc\": rpc error: code = NotFound desc = could not find container \"558ef529829f6719ce73b97c365a08fb4e33709857c8949c7dd42b0104dfd3dc\": container with ID starting with 558ef529829f6719ce73b97c365a08fb4e33709857c8949c7dd42b0104dfd3dc not found: ID does not exist"
Jul 14 15:29:33.701703 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.701680    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f7c8ff86-dbtwv"]
Jul 14 15:29:33.706249 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:33.706225    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67f7c8ff86-dbtwv"]
Jul 14 15:29:34.216401 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:29:34.216370    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88a4571e-bc89-4552-99d9-ac3747bbbec8" path="/var/lib/kubelet/pods/88a4571e-bc89-4552-99d9-ac3747bbbec8/volumes"
Jul 14 15:31:16.063851 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:16.063820    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:31:16.064450 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:16.064429    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:31:30.482916 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.482884    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"]
Jul 14 15:31:30.485428 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.483295    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf84985-4a85-4d60-8ec0-872ca9af256b" containerName="manager"
Jul 14 15:31:30.485428 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.483307    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf84985-4a85-4d60-8ec0-872ca9af256b" containerName="manager"
Jul 14 15:31:30.485428 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.483319    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a4571e-bc89-4552-99d9-ac3747bbbec8" containerName="console"
Jul 14 15:31:30.485428 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.483325    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a4571e-bc89-4552-99d9-ac3747bbbec8" containerName="console"
Jul 14 15:31:30.485428 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.483401    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a4571e-bc89-4552-99d9-ac3747bbbec8" containerName="console"
Jul 14 15:31:30.485428 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.483409    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf84985-4a85-4d60-8ec0-872ca9af256b" containerName="manager"
Jul 14 15:31:30.486349 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.486332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.489082 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.489057    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 14 15:31:30.489306 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.489291    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 14 15:31:30.490622 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.490606    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2rxl9\""
Jul 14 15:31:30.490725 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.490657    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 14 15:31:30.495918 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.495897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"]
Jul 14 15:31:30.617128 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.617102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bff7729f-90fd-4c99-b16d-f61454431316-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.617128 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.617132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bff7729f-90fd-4c99-b16d-f61454431316-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.617353 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.617158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bff7729f-90fd-4c99-b16d-f61454431316-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.617353 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.617227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bff7729f-90fd-4c99-b16d-f61454431316-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.617353 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.617277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bff7729f-90fd-4c99-b16d-f61454431316-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.617353 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.617312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lf97\" (UniqueName: \"kubernetes.io/projected/bff7729f-90fd-4c99-b16d-f61454431316-kube-api-access-2lf97\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.718111 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.718076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bff7729f-90fd-4c99-b16d-f61454431316-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.718111 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.718111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bff7729f-90fd-4c99-b16d-f61454431316-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.718325 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.718135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bff7729f-90fd-4c99-b16d-f61454431316-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.718325 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.718167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lf97\" (UniqueName: \"kubernetes.io/projected/bff7729f-90fd-4c99-b16d-f61454431316-kube-api-access-2lf97\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.718325 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.718264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bff7729f-90fd-4c99-b16d-f61454431316-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.718325 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.718287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bff7729f-90fd-4c99-b16d-f61454431316-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.718611 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.718586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bff7729f-90fd-4c99-b16d-f61454431316-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.718666 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.718611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bff7729f-90fd-4c99-b16d-f61454431316-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.718666 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.718653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bff7729f-90fd-4c99-b16d-f61454431316-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.720744 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.720710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bff7729f-90fd-4c99-b16d-f61454431316-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.720973 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.720958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bff7729f-90fd-4c99-b16d-f61454431316-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.727111 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.727091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lf97\" (UniqueName: \"kubernetes.io/projected/bff7729f-90fd-4c99-b16d-f61454431316-kube-api-access-2lf97\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nfrct\" (UID: \"bff7729f-90fd-4c99-b16d-f61454431316\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.798212 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.798112    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:30.941629 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:30.941592    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"]
Jul 14 15:31:30.943842 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:31:30.943808    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff7729f_90fd_4c99_b16d_f61454431316.slice/crio-3df8c07ff87df5db90d5b2070f43c6d50106f99b0f876697466c4cc5b49fc2f2 WatchSource:0}: Error finding container 3df8c07ff87df5db90d5b2070f43c6d50106f99b0f876697466c4cc5b49fc2f2: Status 404 returned error can't find the container with id 3df8c07ff87df5db90d5b2070f43c6d50106f99b0f876697466c4cc5b49fc2f2
Jul 14 15:31:31.261922 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:31.261878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct" event={"ID":"bff7729f-90fd-4c99-b16d-f61454431316","Type":"ContainerStarted","Data":"3df8c07ff87df5db90d5b2070f43c6d50106f99b0f876697466c4cc5b49fc2f2"}
Jul 14 15:31:32.268098 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:32.268008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct" event={"ID":"bff7729f-90fd-4c99-b16d-f61454431316","Type":"ContainerStarted","Data":"f41cb76edd81ca694cd11c3972d11cfcd1903eb369da8ebe2a83f6e8ccda6f70"}
Jul 14 15:31:32.286129 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:32.286081    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct" podStartSLOduration=1.247848639 podStartE2EDuration="2.286067699s" totalImagesPullingTime="1.03821906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:31:32.283977868 +0000 UTC m=+616.625243788" watchObservedRunningTime="2026-07-14 15:31:32.286067699 +0000 UTC m=+616.627333606"
Jul 14 15:31:40.799296 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:40.799209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:40.799700 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:40.799306    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:40.811847 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:40.811822    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:41.333867 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:41.333837    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nfrct"
Jul 14 15:31:49.580419 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.580380    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"]
Jul 14 15:31:49.586083 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.586049    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.589128 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.589111    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 14 15:31:49.590646 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.590629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.590795 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.590660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.590795 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.590680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.590937 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.590792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.590937 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.590819    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pvl6\" (UniqueName: \"kubernetes.io/projected/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-kube-api-access-8pvl6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.591109 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.590979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.594696 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.594675    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"]
Jul 14 15:31:49.691730 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.691700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.691903 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.691749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.691903 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.691786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.691903 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.691872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.691903 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.691897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pvl6\" (UniqueName: \"kubernetes.io/projected/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-kube-api-access-8pvl6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.692123 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.691977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.692178 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.692162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.692404 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.692380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.692689 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.692606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.693981 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.693961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.694305 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.694288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.701722 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.701682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pvl6\" (UniqueName: \"kubernetes.io/projected/d6b4e6a6-b1b3-44fe-899f-49e8688542b9-kube-api-access-8pvl6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xgb9k\" (UID: \"d6b4e6a6-b1b3-44fe-899f-49e8688542b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:49.898086 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:49.898055    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:50.041794 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:50.041768    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"]
Jul 14 15:31:50.044037 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:31:50.043998    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b4e6a6_b1b3_44fe_899f_49e8688542b9.slice/crio-b3bff9909ce07a7ca03b8cf2361b40bd21b98a49096611f2966549c0bee2b87b WatchSource:0}: Error finding container b3bff9909ce07a7ca03b8cf2361b40bd21b98a49096611f2966549c0bee2b87b: Status 404 returned error can't find the container with id b3bff9909ce07a7ca03b8cf2361b40bd21b98a49096611f2966549c0bee2b87b
Jul 14 15:31:50.045375 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:50.045359    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 15:31:50.365823 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:50.365785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k" event={"ID":"d6b4e6a6-b1b3-44fe-899f-49e8688542b9","Type":"ContainerStarted","Data":"c7ba0dee710195943c9c5f6abf92df65956b443d4ee32a1f56b09e5c201c7940"}
Jul 14 15:31:50.366019 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:50.365845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k" event={"ID":"d6b4e6a6-b1b3-44fe-899f-49e8688542b9","Type":"ContainerStarted","Data":"b3bff9909ce07a7ca03b8cf2361b40bd21b98a49096611f2966549c0bee2b87b"}
Jul 14 15:31:50.384999 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:50.384946    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k" podStartSLOduration=1.167831542 podStartE2EDuration="1.384931077s" totalImagesPullingTime="217.099535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:31:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:31:50.382659771 +0000 UTC m=+634.723925680" watchObservedRunningTime="2026-07-14 15:31:50.384931077 +0000 UTC m=+634.726196984"
Jul 14 15:31:56.775167 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.775130    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"]
Jul 14 15:31:56.779321 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.779305    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.782064 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.782038    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 14 15:31:56.788251 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.788229    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"]
Jul 14 15:31:56.858586 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.858550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.858768 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.858672    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k47wg\" (UniqueName: \"kubernetes.io/projected/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-kube-api-access-k47wg\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.858768 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.858709    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.858768 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.858753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.858901 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.858791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.858901 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.858810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.960207 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.960172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.960207 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.960213    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.960400 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.960352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.960497 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.960479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k47wg\" (UniqueName: \"kubernetes.io/projected/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-kube-api-access-k47wg\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.960586 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.960533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.960586 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.960574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.960586 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.960575    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.960762 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.960742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.960923 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.960904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.962637 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.962606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.963138 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.963117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:56.969212 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:56.969185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k47wg\" (UniqueName: \"kubernetes.io/projected/40eff3ef-ea1d-411c-baea-c23b0ee3aeb9-kube-api-access-k47wg\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44\" (UID: \"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:57.089419 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:57.089323    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:31:57.228899 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:57.228864    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"]
Jul 14 15:31:57.231013 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:31:57.230987    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40eff3ef_ea1d_411c_baea_c23b0ee3aeb9.slice/crio-2705af467ef76286d62d59c514e2f6ab7a6da4bc789817eb566c3b44563da717 WatchSource:0}: Error finding container 2705af467ef76286d62d59c514e2f6ab7a6da4bc789817eb566c3b44563da717: Status 404 returned error can't find the container with id 2705af467ef76286d62d59c514e2f6ab7a6da4bc789817eb566c3b44563da717
Jul 14 15:31:57.404363 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:57.404332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44" event={"ID":"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9","Type":"ContainerStarted","Data":"2705af467ef76286d62d59c514e2f6ab7a6da4bc789817eb566c3b44563da717"}
Jul 14 15:31:58.410657 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:58.410616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44" event={"ID":"40eff3ef-ea1d-411c-baea-c23b0ee3aeb9","Type":"ContainerStarted","Data":"3a7d2d532ab6aae35e0975a9e38e284c95a3fe25d05bc4effa0e9df26b3e8aed"}
Jul 14 15:31:58.434456 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:58.434400    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44" podStartSLOduration=2.223913487 podStartE2EDuration="2.434384238s" totalImagesPullingTime="210.470751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:31:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:31:58.434184161 +0000 UTC m=+642.775450084" watchObservedRunningTime="2026-07-14 15:31:58.434384238 +0000 UTC m=+642.775650144"
Jul 14 15:31:59.898769 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:59.898721    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:59.899282 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:59.898803    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:31:59.911064 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:31:59.911041    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:32:00.436716 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:32:00.436689    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xgb9k"
Jul 14 15:32:07.089854 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:32:07.089815    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:32:07.090291 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:32:07.089880    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:32:07.102942 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:32:07.102907    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:32:07.469402 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:32:07.469373    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44"
Jul 14 15:36:16.096877 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:36:16.096788    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:36:16.100581 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:36:16.100558    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:39:08.684919 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:08.684823    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"]
Jul 14 15:39:08.685546 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:08.685125    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv" podUID="e07be5de-d55a-46dd-b0a8-28674fe9ef6a" containerName="manager" containerID="cri-o://477a4482be4a1b49568f2fb92bff241772ebac77e240dbdb4fb56451b2ab9538" gracePeriod=10
Jul 14 15:39:11.621220 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:11.621128    2563 generic.go:356] "Generic (PLEG): container finished" podID="e07be5de-d55a-46dd-b0a8-28674fe9ef6a" containerID="477a4482be4a1b49568f2fb92bff241772ebac77e240dbdb4fb56451b2ab9538" exitCode=0
Jul 14 15:39:11.621220 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:11.621177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv" event={"ID":"e07be5de-d55a-46dd-b0a8-28674fe9ef6a","Type":"ContainerDied","Data":"477a4482be4a1b49568f2fb92bff241772ebac77e240dbdb4fb56451b2ab9538"}
Jul 14 15:39:11.936739 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:11.936717    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"
Jul 14 15:39:12.064139 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:12.064107    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrhnw\" (UniqueName: \"kubernetes.io/projected/e07be5de-d55a-46dd-b0a8-28674fe9ef6a-kube-api-access-xrhnw\") pod \"e07be5de-d55a-46dd-b0a8-28674fe9ef6a\" (UID: \"e07be5de-d55a-46dd-b0a8-28674fe9ef6a\") "
Jul 14 15:39:12.064330 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:12.064148    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e07be5de-d55a-46dd-b0a8-28674fe9ef6a-extensions-socket-volume\") pod \"e07be5de-d55a-46dd-b0a8-28674fe9ef6a\" (UID: \"e07be5de-d55a-46dd-b0a8-28674fe9ef6a\") "
Jul 14 15:39:12.064550 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:12.064526    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07be5de-d55a-46dd-b0a8-28674fe9ef6a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e07be5de-d55a-46dd-b0a8-28674fe9ef6a" (UID: "e07be5de-d55a-46dd-b0a8-28674fe9ef6a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:39:12.066255 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:12.066223    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07be5de-d55a-46dd-b0a8-28674fe9ef6a-kube-api-access-xrhnw" (OuterVolumeSpecName: "kube-api-access-xrhnw") pod "e07be5de-d55a-46dd-b0a8-28674fe9ef6a" (UID: "e07be5de-d55a-46dd-b0a8-28674fe9ef6a"). InnerVolumeSpecName "kube-api-access-xrhnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:39:12.165486 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:12.165405    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrhnw\" (UniqueName: \"kubernetes.io/projected/e07be5de-d55a-46dd-b0a8-28674fe9ef6a-kube-api-access-xrhnw\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:39:12.165486 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:12.165430    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e07be5de-d55a-46dd-b0a8-28674fe9ef6a-extensions-socket-volume\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:39:12.627310 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:12.627232    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"
Jul 14 15:39:12.627310 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:12.627238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv" event={"ID":"e07be5de-d55a-46dd-b0a8-28674fe9ef6a","Type":"ContainerDied","Data":"c700b506f9340ebfd1686f6a3da1a5cedc9208b8f30872bc4dc50931b17a26ec"}
Jul 14 15:39:12.627310 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:12.627288    2563 scope.go:117] "RemoveContainer" containerID="477a4482be4a1b49568f2fb92bff241772ebac77e240dbdb4fb56451b2ab9538"
Jul 14 15:39:12.648319 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:12.648289    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"]
Jul 14 15:39:12.653264 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:12.653240    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mlfxv"]
Jul 14 15:39:14.223594 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:39:14.223563    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e07be5de-d55a-46dd-b0a8-28674fe9ef6a" path="/var/lib/kubelet/pods/e07be5de-d55a-46dd-b0a8-28674fe9ef6a/volumes"
Jul 14 15:41:16.136533 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:41:16.136487    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:41:16.138901 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:41:16.138883    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:45:00.138749 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:00.138715    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734065-qf89f"]
Jul 14 15:45:00.141083 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:00.139144    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e07be5de-d55a-46dd-b0a8-28674fe9ef6a" containerName="manager"
Jul 14 15:45:00.141083 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:00.139155    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e07be5de-d55a-46dd-b0a8-28674fe9ef6a" containerName="manager"
Jul 14 15:45:00.141083 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:00.139228    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e07be5de-d55a-46dd-b0a8-28674fe9ef6a" containerName="manager"
Jul 14 15:45:00.142394 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:00.142373    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734065-qf89f"
Jul 14 15:45:00.145090 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:00.145071    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\""
Jul 14 15:45:00.152715 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:00.152689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734065-qf89f"
Jul 14 15:45:00.154262 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:00.154238    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734065-qf89f"]
Jul 14 15:45:00.294080 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:00.294052    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734065-qf89f"]
Jul 14 15:45:00.296215 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:45:00.296188    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d5adbd_7d3e_43eb_b900_79f6eb11fc3b.slice/crio-6dd08c51a67aa099b97113a17da1372b3e89f56156041daf72dd084988382371 WatchSource:0}: Error finding container 6dd08c51a67aa099b97113a17da1372b3e89f56156041daf72dd084988382371: Status 404 returned error can't find the container with id 6dd08c51a67aa099b97113a17da1372b3e89f56156041daf72dd084988382371
Jul 14 15:45:00.297989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:00.297971    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 15:45:00.358046 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:00.358010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734065-qf89f" event={"ID":"f9d5adbd-7d3e-43eb-b900-79f6eb11fc3b","Type":"ContainerStarted","Data":"6dd08c51a67aa099b97113a17da1372b3e89f56156041daf72dd084988382371"}
Jul 14 15:45:04.380454 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:04.380356    2563 generic.go:356] "Generic (PLEG): container finished" podID="f9d5adbd-7d3e-43eb-b900-79f6eb11fc3b" containerID="8a3be6542db79a3de117c5062e79f352e36887e2e0d9a0da739ffe60c3c2c990" exitCode=0
Jul 14 15:45:04.380895 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:04.380451    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734065-qf89f" event={"ID":"f9d5adbd-7d3e-43eb-b900-79f6eb11fc3b","Type":"ContainerDied","Data":"8a3be6542db79a3de117c5062e79f352e36887e2e0d9a0da739ffe60c3c2c990"}
Jul 14 15:45:05.518833 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:05.518812    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734065-qf89f"
Jul 14 15:45:06.391719 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:06.391688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734065-qf89f" event={"ID":"f9d5adbd-7d3e-43eb-b900-79f6eb11fc3b","Type":"ContainerDied","Data":"6dd08c51a67aa099b97113a17da1372b3e89f56156041daf72dd084988382371"}
Jul 14 15:45:06.391875 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:06.391734    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dd08c51a67aa099b97113a17da1372b3e89f56156041daf72dd084988382371"
Jul 14 15:45:06.391875 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:45:06.391733    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734065-qf89f"
Jul 14 15:46:16.168773 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:16.168739    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:46:16.172699 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:16.172676    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:46:19.270767 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.270706    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"]
Jul 14 15:46:19.271590 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.271563    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9d5adbd-7d3e-43eb-b900-79f6eb11fc3b" containerName="cleanup"
Jul 14 15:46:19.271590 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.271592    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9d5adbd-7d3e-43eb-b900-79f6eb11fc3b" containerName="cleanup"
Jul 14 15:46:19.271772 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.271725    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9d5adbd-7d3e-43eb-b900-79f6eb11fc3b" containerName="cleanup"
Jul 14 15:46:19.274865 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.274839    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.277865 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.277838    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 15:46:19.277966 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.277873    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 15:46:19.278032 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.277994    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-94af7c4d-openshift-default-dockercfg-rhpm5\""
Jul 14 15:46:19.278150 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.278134    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 15:46:19.287384 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.287188    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"]
Jul 14 15:46:19.403466 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.403427    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4001f1a-d506-4f09-ac97-bef509545c70-istiod-ca-cert\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.403466 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.403466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4001f1a-d506-4f09-ac97-bef509545c70-istio-token\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.403720 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.403525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-workload-socket\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.403720 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.403582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-workload-certs\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.403720 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.403624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4001f1a-d506-4f09-ac97-bef509545c70-istio-podinfo\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.403720 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.403645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-credential-socket\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.403720 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.403674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-istio-envoy\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.403879 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.403757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-istio-data\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.403879 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.403809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrf6\" (UniqueName: \"kubernetes.io/projected/d4001f1a-d506-4f09-ac97-bef509545c70-kube-api-access-mjrf6\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.504610 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.504569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-istio-data\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.504789 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.504621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrf6\" (UniqueName: \"kubernetes.io/projected/d4001f1a-d506-4f09-ac97-bef509545c70-kube-api-access-mjrf6\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.504789 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.504654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4001f1a-d506-4f09-ac97-bef509545c70-istiod-ca-cert\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.504789 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.504672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4001f1a-d506-4f09-ac97-bef509545c70-istio-token\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.504789 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.504702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-workload-socket\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.504789 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.504725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-workload-certs\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.504789 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.504765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4001f1a-d506-4f09-ac97-bef509545c70-istio-podinfo\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.505122 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.504791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-credential-socket\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.505122 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.504824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-istio-envoy\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.505122 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.505018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-istio-data\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.505280 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.505191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-workload-certs\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.505280 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.505233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-credential-socket\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.505429 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.505352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4001f1a-d506-4f09-ac97-bef509545c70-istiod-ca-cert\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.505484 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.505446    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-workload-socket\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.507221 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.507197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-istio-envoy\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.507416 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.507398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4001f1a-d506-4f09-ac97-bef509545c70-istio-podinfo\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.514787 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.514752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrf6\" (UniqueName: \"kubernetes.io/projected/d4001f1a-d506-4f09-ac97-bef509545c70-kube-api-access-mjrf6\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.515538 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.515495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4001f1a-d506-4f09-ac97-bef509545c70-istio-token\") pod \"e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") " pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.588026 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.587939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:19.738752 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.738720    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"]
Jul 14 15:46:19.740210 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:46:19.740181    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4001f1a_d506_4f09_ac97_bef509545c70.slice/crio-e354d8d730da57dc86496500c8657761ef751bf86fde2fc30668f3b1478ef81e WatchSource:0}: Error finding container e354d8d730da57dc86496500c8657761ef751bf86fde2fc30668f3b1478ef81e: Status 404 returned error can't find the container with id e354d8d730da57dc86496500c8657761ef751bf86fde2fc30668f3b1478ef81e
Jul 14 15:46:19.765780 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:19.765746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj" event={"ID":"d4001f1a-d506-4f09-ac97-bef509545c70","Type":"ContainerStarted","Data":"e354d8d730da57dc86496500c8657761ef751bf86fde2fc30668f3b1478ef81e"}
Jul 14 15:46:20.596780 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.596730    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"]
Jul 14 15:46:20.602074 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.602052    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:46:20.604953 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.604926    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-94af7c4d-serving-cert\""
Jul 14 15:46:20.605074 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.604991    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 15:46:20.605172 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.605147    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 15:46:20.606263 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.606241    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npnj7\""
Jul 14 15:46:20.611280 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.610944    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"]
Jul 14 15:46:20.717078 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.717046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq9pz\" (UniqueName: \"kubernetes.io/projected/0f1eb069-5dea-4edf-a60e-65eaa74c173e-kube-api-access-lq9pz\") pod \"maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2\" (UID: \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:46:20.717245 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.717135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1eb069-5dea-4edf-a60e-65eaa74c173e-openshift-service-ca\") pod \"maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2\" (UID: \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:46:20.717245 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.717175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f1eb069-5dea-4edf-a60e-65eaa74c173e-maas-api-tls\") pod \"maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2\" (UID: \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:46:20.817997 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.817963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1eb069-5dea-4edf-a60e-65eaa74c173e-openshift-service-ca\") pod \"maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2\" (UID: \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:46:20.818189 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.818037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f1eb069-5dea-4edf-a60e-65eaa74c173e-maas-api-tls\") pod \"maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2\" (UID: \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:46:20.818189 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.818076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq9pz\" (UniqueName: \"kubernetes.io/projected/0f1eb069-5dea-4edf-a60e-65eaa74c173e-kube-api-access-lq9pz\") pod \"maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2\" (UID: \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:46:20.818189 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:46:20.818147    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-serving-cert: secret "maas-api-e2e-ait-94af7c4d-serving-cert" not found
Jul 14 15:46:20.818334 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:46:20.818250    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f1eb069-5dea-4edf-a60e-65eaa74c173e-maas-api-tls podName:0f1eb069-5dea-4edf-a60e-65eaa74c173e nodeName:}" failed. No retries permitted until 2026-07-14 15:46:21.318233513 +0000 UTC m=+1505.659499399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f1eb069-5dea-4edf-a60e-65eaa74c173e-maas-api-tls") pod "maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2" (UID: "0f1eb069-5dea-4edf-a60e-65eaa74c173e") : secret "maas-api-e2e-ait-94af7c4d-serving-cert" not found
Jul 14 15:46:20.818702 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.818686    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1eb069-5dea-4edf-a60e-65eaa74c173e-openshift-service-ca\") pod \"maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2\" (UID: \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:46:20.830286 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:20.830262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq9pz\" (UniqueName: \"kubernetes.io/projected/0f1eb069-5dea-4edf-a60e-65eaa74c173e-kube-api-access-lq9pz\") pod \"maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2\" (UID: \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:46:21.323218 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:21.323181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f1eb069-5dea-4edf-a60e-65eaa74c173e-maas-api-tls\") pod \"maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2\" (UID: \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:46:21.325833 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:21.325804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f1eb069-5dea-4edf-a60e-65eaa74c173e-maas-api-tls\") pod \"maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2\" (UID: \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:46:21.515340 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:21.515303    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:46:21.651766 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:21.651738    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"]
Jul 14 15:46:21.653173 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:46:21.653145    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1eb069_5dea_4edf_a60e_65eaa74c173e.slice/crio-5680005cdf5779f2167d3a4ac1fc7ad5f584d8d50ab68d096d3b0c636521ba28 WatchSource:0}: Error finding container 5680005cdf5779f2167d3a4ac1fc7ad5f584d8d50ab68d096d3b0c636521ba28: Status 404 returned error can't find the container with id 5680005cdf5779f2167d3a4ac1fc7ad5f584d8d50ab68d096d3b0c636521ba28
Jul 14 15:46:21.782033 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:21.781992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2" event={"ID":"0f1eb069-5dea-4edf-a60e-65eaa74c173e","Type":"ContainerStarted","Data":"5680005cdf5779f2167d3a4ac1fc7ad5f584d8d50ab68d096d3b0c636521ba28"}
Jul 14 15:46:23.878907 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:23.878868    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:46:23.879177 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:23.878942    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:46:23.879177 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:23.878972    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:46:24.801684 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:24.801643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj" event={"ID":"d4001f1a-d506-4f09-ac97-bef509545c70","Type":"ContainerStarted","Data":"83b85e4ddb480a0629617042302b580974667d2ee55897a883250a160a8fd87d"}
Jul 14 15:46:24.836097 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:24.836031    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj" podStartSLOduration=1.69963263 podStartE2EDuration="5.83600871s" totalImagesPullingTime="4.13637608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:46:24.831719212 +0000 UTC m=+1509.172985154" watchObservedRunningTime="2026-07-14 15:46:24.83600871 +0000 UTC m=+1509.177274620"
Jul 14 15:46:25.577456 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:25.577420    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4"]
Jul 14 15:46:25.581135 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:25.581118    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4"
Jul 14 15:46:25.583761 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:25.583738    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\""
Jul 14 15:46:25.588080 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:25.588063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:25.589868 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:25.589842    2563 patch_prober.go:28] interesting pod/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 14 15:46:25.589958 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:25.589898    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj" podUID="d4001f1a-d506-4f09-ac97-bef509545c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 14 15:46:25.592433 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:25.592412    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4"]
Jul 14 15:46:25.670630 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:25.670592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f25a4e0-e498-42c3-a927-785e70dfa413-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4\" (UID: \"9f25a4e0-e498-42c3-a927-785e70dfa413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4"
Jul 14 15:46:25.772227 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:25.772180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f25a4e0-e498-42c3-a927-785e70dfa413-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4\" (UID: \"9f25a4e0-e498-42c3-a927-785e70dfa413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4"
Jul 14 15:46:25.772828 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:25.772806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f25a4e0-e498-42c3-a927-785e70dfa413-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4\" (UID: \"9f25a4e0-e498-42c3-a927-785e70dfa413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4"
Jul 14 15:46:25.808065 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:25.808033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2" event={"ID":"0f1eb069-5dea-4edf-a60e-65eaa74c173e","Type":"ContainerStarted","Data":"1a7ef19f3fafd53ef7c79645a7410917690ea16032d2113b29de38aa2646eb34"}
Jul 14 15:46:25.808208 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:25.808140    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:46:25.892417 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:25.892377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4"
Jul 14 15:46:26.084299 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:26.084251    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2" podStartSLOduration=2.362091832 podStartE2EDuration="6.084233807s" totalImagesPullingTime="3.722141975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:46:25.828837812 +0000 UTC m=+1510.170103738" watchObservedRunningTime="2026-07-14 15:46:26.084233807 +0000 UTC m=+1510.425499716"
Jul 14 15:46:26.085846 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:26.085827    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4"]
Jul 14 15:46:26.088564 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:46:26.088537    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f25a4e0_e498_42c3_a927_785e70dfa413.slice/crio-74e63ee0e1b762e389390528b903e86aca7deb188e8d56bba02d468b74dee2d3 WatchSource:0}: Error finding container 74e63ee0e1b762e389390528b903e86aca7deb188e8d56bba02d468b74dee2d3: Status 404 returned error can't find the container with id 74e63ee0e1b762e389390528b903e86aca7deb188e8d56bba02d468b74dee2d3
Jul 14 15:46:26.588743 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:26.588700    2563 patch_prober.go:28] interesting pod/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 14 15:46:26.589118 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:26.588789    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj" podUID="d4001f1a-d506-4f09-ac97-bef509545c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 14 15:46:26.814905 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:26.814871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4" event={"ID":"9f25a4e0-e498-42c3-a927-785e70dfa413","Type":"ContainerStarted","Data":"9211370020fb7b24ba0717e81d26a86f21416df11abdd477a534b6f3c0d69b19"}
Jul 14 15:46:26.815073 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:26.814923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4" event={"ID":"9f25a4e0-e498-42c3-a927-785e70dfa413","Type":"ContainerStarted","Data":"74e63ee0e1b762e389390528b903e86aca7deb188e8d56bba02d468b74dee2d3"}
Jul 14 15:46:26.834533 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:26.832327    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4" podStartSLOduration=1.8323052560000002 podStartE2EDuration="1.832305256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:46:26.831312648 +0000 UTC m=+1511.172578556" watchObservedRunningTime="2026-07-14 15:46:26.832305256 +0000 UTC m=+1511.173571164"
Jul 14 15:46:27.589175 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:27.589134    2563 patch_prober.go:28] interesting pod/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 14 15:46:27.589664 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:27.589200    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj" podUID="d4001f1a-d506-4f09-ac97-bef509545c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 14 15:46:27.822535 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:27.822476    2563 generic.go:356] "Generic (PLEG): container finished" podID="9f25a4e0-e498-42c3-a927-785e70dfa413" containerID="9211370020fb7b24ba0717e81d26a86f21416df11abdd477a534b6f3c0d69b19" exitCode=7
Jul 14 15:46:27.822738 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:27.822607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4" event={"ID":"9f25a4e0-e498-42c3-a927-785e70dfa413","Type":"ContainerDied","Data":"9211370020fb7b24ba0717e81d26a86f21416df11abdd477a534b6f3c0d69b19"}
Jul 14 15:46:27.822970 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:27.822947    2563 scope.go:117] "RemoveContainer" containerID="9211370020fb7b24ba0717e81d26a86f21416df11abdd477a534b6f3c0d69b19"
Jul 14 15:46:28.589068 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:28.588971    2563 patch_prober.go:28] interesting pod/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 14 15:46:28.589068 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:28.589040    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj" podUID="d4001f1a-d506-4f09-ac97-bef509545c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 14 15:46:28.833621 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:28.833581    2563 generic.go:356] "Generic (PLEG): container finished" podID="9f25a4e0-e498-42c3-a927-785e70dfa413" containerID="0b201d91f3bb4e72afe1403d189720b0f5579e6c4ebf5b3a99ce8aeeecec31ee" exitCode=7
Jul 14 15:46:28.833816 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:28.833715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4" event={"ID":"9f25a4e0-e498-42c3-a927-785e70dfa413","Type":"ContainerDied","Data":"0b201d91f3bb4e72afe1403d189720b0f5579e6c4ebf5b3a99ce8aeeecec31ee"}
Jul 14 15:46:28.833816 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:28.833774    2563 scope.go:117] "RemoveContainer" containerID="9211370020fb7b24ba0717e81d26a86f21416df11abdd477a534b6f3c0d69b19"
Jul 14 15:46:28.834213 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:28.834190    2563 scope.go:117] "RemoveContainer" containerID="0b201d91f3bb4e72afe1403d189720b0f5579e6c4ebf5b3a99ce8aeeecec31ee"
Jul 14 15:46:28.834591 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:46:28.834476    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4_odh-ai-gateway-infra(9f25a4e0-e498-42c3-a927-785e70dfa413)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4" podUID="9f25a4e0-e498-42c3-a927-785e70dfa413"
Jul 14 15:46:29.592810 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:29.592734    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:29.593174 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:29.592989    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:29.593696 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:29.593677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:46:29.850135 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:29.850049    2563 scope.go:117] "RemoveContainer" containerID="0b201d91f3bb4e72afe1403d189720b0f5579e6c4ebf5b3a99ce8aeeecec31ee"
Jul 14 15:46:29.850301 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:46:29.850278    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4_odh-ai-gateway-infra(9f25a4e0-e498-42c3-a927-785e70dfa413)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4" podUID="9f25a4e0-e498-42c3-a927-785e70dfa413"
Jul 14 15:46:31.821183 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:31.821154    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:46:41.211191 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:41.211089    2563 scope.go:117] "RemoveContainer" containerID="0b201d91f3bb4e72afe1403d189720b0f5579e6c4ebf5b3a99ce8aeeecec31ee"
Jul 14 15:46:41.913655 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:41.913602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4" event={"ID":"9f25a4e0-e498-42c3-a927-785e70dfa413","Type":"ContainerStarted","Data":"d2361754d35410d674ae35d719d9869b79759ee152a36476f0df3ff932b9191b"}
Jul 14 15:46:42.921125 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:42.921081    2563 generic.go:356] "Generic (PLEG): container finished" podID="9f25a4e0-e498-42c3-a927-785e70dfa413" containerID="d2361754d35410d674ae35d719d9869b79759ee152a36476f0df3ff932b9191b" exitCode=0
Jul 14 15:46:42.921562 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:42.921155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4" event={"ID":"9f25a4e0-e498-42c3-a927-785e70dfa413","Type":"ContainerDied","Data":"d2361754d35410d674ae35d719d9869b79759ee152a36476f0df3ff932b9191b"}
Jul 14 15:46:42.921562 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:42.921217    2563 scope.go:117] "RemoveContainer" containerID="0b201d91f3bb4e72afe1403d189720b0f5579e6c4ebf5b3a99ce8aeeecec31ee"
Jul 14 15:46:42.948665 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:42.948631    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4"]
Jul 14 15:46:44.057290 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:44.057263    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4"
Jul 14 15:46:44.156965 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:44.156933    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f25a4e0-e498-42c3-a927-785e70dfa413-maas-api-service-ca\") pod \"9f25a4e0-e498-42c3-a927-785e70dfa413\" (UID: \"9f25a4e0-e498-42c3-a927-785e70dfa413\") "
Jul 14 15:46:44.157288 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:44.157266    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f25a4e0-e498-42c3-a927-785e70dfa413-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f25a4e0-e498-42c3-a927-785e70dfa413" (UID: "9f25a4e0-e498-42c3-a927-785e70dfa413"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:46:44.258203 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:44.258171    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f25a4e0-e498-42c3-a927-785e70dfa413-maas-api-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:46:44.933836 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:44.933807    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4"
Jul 14 15:46:44.933993 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:44.933848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4" event={"ID":"9f25a4e0-e498-42c3-a927-785e70dfa413","Type":"ContainerDied","Data":"74e63ee0e1b762e389390528b903e86aca7deb188e8d56bba02d468b74dee2d3"}
Jul 14 15:46:44.933993 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:44.933902    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74e63ee0e1b762e389390528b903e86aca7deb188e8d56bba02d468b74dee2d3"
Jul 14 15:46:44.971254 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:44.971219    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4"]
Jul 14 15:46:44.975585 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:44.975561    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-qjxc4"]
Jul 14 15:46:46.217377 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:46:46.217345    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f25a4e0-e498-42c3-a927-785e70dfa413" path="/var/lib/kubelet/pods/9f25a4e0-e498-42c3-a927-785e70dfa413/volumes"
Jul 14 15:47:15.632250 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:15.632214    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx"]
Jul 14 15:47:15.632699 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:15.632663    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f25a4e0-e498-42c3-a927-785e70dfa413" containerName="revoke-keys"
Jul 14 15:47:15.632699 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:15.632675    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f25a4e0-e498-42c3-a927-785e70dfa413" containerName="revoke-keys"
Jul 14 15:47:15.632699 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:15.632697    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f25a4e0-e498-42c3-a927-785e70dfa413" containerName="revoke-keys"
Jul 14 15:47:15.632803 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:15.632702    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f25a4e0-e498-42c3-a927-785e70dfa413" containerName="revoke-keys"
Jul 14 15:47:15.632803 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:15.632766    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f25a4e0-e498-42c3-a927-785e70dfa413" containerName="revoke-keys"
Jul 14 15:47:15.632803 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:15.632778    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f25a4e0-e498-42c3-a927-785e70dfa413" containerName="revoke-keys"
Jul 14 15:47:15.635050 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:15.635035    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx"
Jul 14 15:47:15.637845 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:15.637825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\""
Jul 14 15:47:15.643004 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:15.642978    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx"]
Jul 14 15:47:15.773095 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:15.773051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e79ea4b9-d059-44f2-894e-4b7b0785e4d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx\" (UID: \"e79ea4b9-d059-44f2-894e-4b7b0785e4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx"
Jul 14 15:47:15.874579 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:15.874540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e79ea4b9-d059-44f2-894e-4b7b0785e4d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx\" (UID: \"e79ea4b9-d059-44f2-894e-4b7b0785e4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx"
Jul 14 15:47:15.875134 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:15.875114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e79ea4b9-d059-44f2-894e-4b7b0785e4d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx\" (UID: \"e79ea4b9-d059-44f2-894e-4b7b0785e4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx"
Jul 14 15:47:15.946426 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:15.946353    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx"
Jul 14 15:47:16.076154 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:16.076132    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx"]
Jul 14 15:47:16.078219 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:47:16.078196    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79ea4b9_d059_44f2_894e_4b7b0785e4d5.slice/crio-3ee455c04e840cd4b258a43207e13599eba1be10a3f227aea3b81cc410e61b6e WatchSource:0}: Error finding container 3ee455c04e840cd4b258a43207e13599eba1be10a3f227aea3b81cc410e61b6e: Status 404 returned error can't find the container with id 3ee455c04e840cd4b258a43207e13599eba1be10a3f227aea3b81cc410e61b6e
Jul 14 15:47:16.096020 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:16.095994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx" event={"ID":"e79ea4b9-d059-44f2-894e-4b7b0785e4d5","Type":"ContainerStarted","Data":"3ee455c04e840cd4b258a43207e13599eba1be10a3f227aea3b81cc410e61b6e"}
Jul 14 15:47:17.103681 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:17.103647    2563 generic.go:356] "Generic (PLEG): container finished" podID="e79ea4b9-d059-44f2-894e-4b7b0785e4d5" containerID="64a309460e9b4013f2421d138ebd2391622ec30078fc508b57b30f2b5ad1b626" exitCode=0
Jul 14 15:47:17.104127 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:17.103727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx" event={"ID":"e79ea4b9-d059-44f2-894e-4b7b0785e4d5","Type":"ContainerDied","Data":"64a309460e9b4013f2421d138ebd2391622ec30078fc508b57b30f2b5ad1b626"}
Jul 14 15:47:18.248674 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:18.248651    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx"
Jul 14 15:47:18.400290 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:18.400262    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e79ea4b9-d059-44f2-894e-4b7b0785e4d5-maas-api-service-ca\") pod \"e79ea4b9-d059-44f2-894e-4b7b0785e4d5\" (UID: \"e79ea4b9-d059-44f2-894e-4b7b0785e4d5\") "
Jul 14 15:47:18.400607 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:18.400586    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79ea4b9-d059-44f2-894e-4b7b0785e4d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e79ea4b9-d059-44f2-894e-4b7b0785e4d5" (UID: "e79ea4b9-d059-44f2-894e-4b7b0785e4d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:47:18.501137 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:18.501101    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e79ea4b9-d059-44f2-894e-4b7b0785e4d5-maas-api-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:47:19.115073 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:19.115035    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx"
Jul 14 15:47:19.115073 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:19.115040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx" event={"ID":"e79ea4b9-d059-44f2-894e-4b7b0785e4d5","Type":"ContainerDied","Data":"3ee455c04e840cd4b258a43207e13599eba1be10a3f227aea3b81cc410e61b6e"}
Jul 14 15:47:19.115296 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:19.115126    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ee455c04e840cd4b258a43207e13599eba1be10a3f227aea3b81cc410e61b6e"
Jul 14 15:47:26.306481 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:26.306447    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"]
Jul 14 15:47:26.307027 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:26.306725    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2" podUID="0f1eb069-5dea-4edf-a60e-65eaa74c173e" containerName="maas-api" containerID="cri-o://1a7ef19f3fafd53ef7c79645a7410917690ea16032d2113b29de38aa2646eb34" gracePeriod=30
Jul 14 15:47:26.557634 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:26.557575    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:47:26.680945 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:26.680896    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f1eb069-5dea-4edf-a60e-65eaa74c173e-maas-api-tls\") pod \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\" (UID: \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\") "
Jul 14 15:47:26.681132 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:26.681003    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1eb069-5dea-4edf-a60e-65eaa74c173e-openshift-service-ca\") pod \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\" (UID: \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\") "
Jul 14 15:47:26.681132 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:26.681065    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq9pz\" (UniqueName: \"kubernetes.io/projected/0f1eb069-5dea-4edf-a60e-65eaa74c173e-kube-api-access-lq9pz\") pod \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\" (UID: \"0f1eb069-5dea-4edf-a60e-65eaa74c173e\") "
Jul 14 15:47:26.681371 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:26.681343    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1eb069-5dea-4edf-a60e-65eaa74c173e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f1eb069-5dea-4edf-a60e-65eaa74c173e" (UID: "0f1eb069-5dea-4edf-a60e-65eaa74c173e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:47:26.683316 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:26.683285    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1eb069-5dea-4edf-a60e-65eaa74c173e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f1eb069-5dea-4edf-a60e-65eaa74c173e" (UID: "0f1eb069-5dea-4edf-a60e-65eaa74c173e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:47:26.683316 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:26.683295    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1eb069-5dea-4edf-a60e-65eaa74c173e-kube-api-access-lq9pz" (OuterVolumeSpecName: "kube-api-access-lq9pz") pod "0f1eb069-5dea-4edf-a60e-65eaa74c173e" (UID: "0f1eb069-5dea-4edf-a60e-65eaa74c173e"). InnerVolumeSpecName "kube-api-access-lq9pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:47:26.781779 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:26.781744    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1eb069-5dea-4edf-a60e-65eaa74c173e-openshift-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:47:26.781779 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:26.781770    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq9pz\" (UniqueName: \"kubernetes.io/projected/0f1eb069-5dea-4edf-a60e-65eaa74c173e-kube-api-access-lq9pz\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:47:26.781779 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:26.781779    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f1eb069-5dea-4edf-a60e-65eaa74c173e-maas-api-tls\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:47:27.157429 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:27.157399    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f1eb069-5dea-4edf-a60e-65eaa74c173e" containerID="1a7ef19f3fafd53ef7c79645a7410917690ea16032d2113b29de38aa2646eb34" exitCode=0
Jul 14 15:47:27.157632 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:27.157445    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"
Jul 14 15:47:27.157632 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:27.157471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2" event={"ID":"0f1eb069-5dea-4edf-a60e-65eaa74c173e","Type":"ContainerDied","Data":"1a7ef19f3fafd53ef7c79645a7410917690ea16032d2113b29de38aa2646eb34"}
Jul 14 15:47:27.157632 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:27.157539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2" event={"ID":"0f1eb069-5dea-4edf-a60e-65eaa74c173e","Type":"ContainerDied","Data":"5680005cdf5779f2167d3a4ac1fc7ad5f584d8d50ab68d096d3b0c636521ba28"}
Jul 14 15:47:27.157632 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:27.157574    2563 scope.go:117] "RemoveContainer" containerID="1a7ef19f3fafd53ef7c79645a7410917690ea16032d2113b29de38aa2646eb34"
Jul 14 15:47:27.166741 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:27.166723    2563 scope.go:117] "RemoveContainer" containerID="1a7ef19f3fafd53ef7c79645a7410917690ea16032d2113b29de38aa2646eb34"
Jul 14 15:47:27.166975 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:47:27.166958    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7ef19f3fafd53ef7c79645a7410917690ea16032d2113b29de38aa2646eb34\": container with ID starting with 1a7ef19f3fafd53ef7c79645a7410917690ea16032d2113b29de38aa2646eb34 not found: ID does not exist" containerID="1a7ef19f3fafd53ef7c79645a7410917690ea16032d2113b29de38aa2646eb34"
Jul 14 15:47:27.167044 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:27.166984    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7ef19f3fafd53ef7c79645a7410917690ea16032d2113b29de38aa2646eb34"} err="failed to get container status \"1a7ef19f3fafd53ef7c79645a7410917690ea16032d2113b29de38aa2646eb34\": rpc error: code = NotFound desc = could not find container \"1a7ef19f3fafd53ef7c79645a7410917690ea16032d2113b29de38aa2646eb34\": container with ID starting with 1a7ef19f3fafd53ef7c79645a7410917690ea16032d2113b29de38aa2646eb34 not found: ID does not exist"
Jul 14 15:47:27.180865 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:27.180820    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"]
Jul 14 15:47:27.183093 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:27.183073    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-94af7c4d-5bbf468744-8lmd2"]
Jul 14 15:47:28.215820 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:28.215791    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f1eb069-5dea-4edf-a60e-65eaa74c173e" path="/var/lib/kubelet/pods/0f1eb069-5dea-4edf-a60e-65eaa74c173e/volumes"
Jul 14 15:47:40.948853 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:40.948818    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"]
Jul 14 15:47:40.949274 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:40.949013    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj" podUID="d4001f1a-d506-4f09-ac97-bef509545c70" containerName="istio-proxy" containerID="cri-o://83b85e4ddb480a0629617042302b580974667d2ee55897a883250a160a8fd87d" gracePeriod=30
Jul 14 15:47:41.557530 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.557467    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"]
Jul 14 15:47:41.557942 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.557928    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1eb069-5dea-4edf-a60e-65eaa74c173e" containerName="maas-api"
Jul 14 15:47:41.557991 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.557945    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1eb069-5dea-4edf-a60e-65eaa74c173e" containerName="maas-api"
Jul 14 15:47:41.557991 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.557962    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f25a4e0-e498-42c3-a927-785e70dfa413" containerName="revoke-keys"
Jul 14 15:47:41.557991 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.557967    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f25a4e0-e498-42c3-a927-785e70dfa413" containerName="revoke-keys"
Jul 14 15:47:41.557991 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.557989    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e79ea4b9-d059-44f2-894e-4b7b0785e4d5" containerName="revoke-keys"
Jul 14 15:47:41.558111 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.557996    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e79ea4b9-d059-44f2-894e-4b7b0785e4d5" containerName="revoke-keys"
Jul 14 15:47:41.558111 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.558054    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1eb069-5dea-4edf-a60e-65eaa74c173e" containerName="maas-api"
Jul 14 15:47:41.558111 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.558061    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e79ea4b9-d059-44f2-894e-4b7b0785e4d5" containerName="revoke-keys"
Jul 14 15:47:41.558111 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.558070    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f25a4e0-e498-42c3-a927-785e70dfa413" containerName="revoke-keys"
Jul 14 15:47:41.562371 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.562348    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.565065 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.565047    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-65fd3bf5-gw-openshift-default-dockercfg-btn8f\""
Jul 14 15:47:41.574492 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.574473    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"]
Jul 14 15:47:41.621944 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.621913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-workload-socket\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.622099 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.621958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef7abe83-94f1-41f5-ac58-e66792293425-istiod-ca-cert\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.622099 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.621985    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-workload-certs\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.622099 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.622056    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smlh7\" (UniqueName: \"kubernetes.io/projected/ef7abe83-94f1-41f5-ac58-e66792293425-kube-api-access-smlh7\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.622264 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.622145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-istio-envoy\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.622264 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.622173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef7abe83-94f1-41f5-ac58-e66792293425-istio-podinfo\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.622264 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.622198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef7abe83-94f1-41f5-ac58-e66792293425-istio-token\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.622264 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.622234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-credential-socket\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.622439 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.622279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-istio-data\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.723378 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.723345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-istio-envoy\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.723378 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.723376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef7abe83-94f1-41f5-ac58-e66792293425-istio-podinfo\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.723649 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.723396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef7abe83-94f1-41f5-ac58-e66792293425-istio-token\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.723649 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.723419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-credential-socket\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.723649 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.723439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-istio-data\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.723649 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.723542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-workload-socket\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.723649 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.723580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef7abe83-94f1-41f5-ac58-e66792293425-istiod-ca-cert\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.723649 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.723604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-workload-certs\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.723649 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.723629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smlh7\" (UniqueName: \"kubernetes.io/projected/ef7abe83-94f1-41f5-ac58-e66792293425-kube-api-access-smlh7\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.724032 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.723883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-credential-socket\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.724032 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.723989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-istio-data\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.724183 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.724161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-workload-certs\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.724293 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.724275    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef7abe83-94f1-41f5-ac58-e66792293425-istiod-ca-cert\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.724399 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.724377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-workload-socket\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.725794 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.725773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-istio-envoy\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.726061 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.726042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef7abe83-94f1-41f5-ac58-e66792293425-istio-podinfo\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.734081 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.734034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef7abe83-94f1-41f5-ac58-e66792293425-istio-token\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.734346 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.734329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smlh7\" (UniqueName: \"kubernetes.io/projected/ef7abe83-94f1-41f5-ac58-e66792293425-kube-api-access-smlh7\") pod \"e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") " pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:41.874378 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:41.874287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:42.020611 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:42.020570    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"]
Jul 14 15:47:42.022202 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:47:42.022172    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7abe83_94f1_41f5_ac58_e66792293425.slice/crio-b194ba0cb5e2c1f5643afda484aac6fdb3a74ff5cd5d495b2bf2614ab454daa2 WatchSource:0}: Error finding container b194ba0cb5e2c1f5643afda484aac6fdb3a74ff5cd5d495b2bf2614ab454daa2: Status 404 returned error can't find the container with id b194ba0cb5e2c1f5643afda484aac6fdb3a74ff5cd5d495b2bf2614ab454daa2
Jul 14 15:47:42.032913 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:42.032868    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:47:42.033017 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:42.032957    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:47:42.033017 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:42.032989    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:47:42.239603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:42.239572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6" event={"ID":"ef7abe83-94f1-41f5-ac58-e66792293425","Type":"ContainerStarted","Data":"325a099d729794602d6e888487338e57eaccc43394b0856be5f7ad229ce7864b"}
Jul 14 15:47:42.239780 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:42.239630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6" event={"ID":"ef7abe83-94f1-41f5-ac58-e66792293425","Type":"ContainerStarted","Data":"b194ba0cb5e2c1f5643afda484aac6fdb3a74ff5cd5d495b2bf2614ab454daa2"}
Jul 14 15:47:42.264434 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:42.264383    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6" podStartSLOduration=1.264366741 podStartE2EDuration="1.264366741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:47:42.262969662 +0000 UTC m=+1586.604235596" watchObservedRunningTime="2026-07-14 15:47:42.264366741 +0000 UTC m=+1586.605632647"
Jul 14 15:47:42.875098 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:42.875063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:43.094862 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.094829    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"]
Jul 14 15:47:43.098935 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.098918    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:47:43.102792 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.102768    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 15:47:43.102898 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.102842    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-65fd3bf5-serving-cert\""
Jul 14 15:47:43.104141 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.104125    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npnj7\""
Jul 14 15:47:43.104231 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.104180    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 15:47:43.109921 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.109899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"]
Jul 14 15:47:43.142134 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.142106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-openshift-service-ca\") pod \"maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc\" (UID: \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:47:43.142260 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.142229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-maas-api-tls\") pod \"maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc\" (UID: \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:47:43.142314 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.142274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2wpq\" (UniqueName: \"kubernetes.io/projected/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-kube-api-access-s2wpq\") pod \"maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc\" (UID: \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:47:43.243099 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.243072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-openshift-service-ca\") pod \"maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc\" (UID: \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:47:43.243255 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.243168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-maas-api-tls\") pod \"maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc\" (UID: \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:47:43.243255 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.243241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2wpq\" (UniqueName: \"kubernetes.io/projected/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-kube-api-access-s2wpq\") pod \"maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc\" (UID: \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:47:43.243390 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:47:43.243325    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-serving-cert: secret "maas-api-e2e-migrate-65fd3bf5-serving-cert" not found
Jul 14 15:47:43.243450 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:47:43.243397    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-maas-api-tls podName:63ff61f1-75b5-4d66-b63a-dbc4bc73af26 nodeName:}" failed. No retries permitted until 2026-07-14 15:47:43.743373399 +0000 UTC m=+1588.084639307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-maas-api-tls") pod "maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc" (UID: "63ff61f1-75b5-4d66-b63a-dbc4bc73af26") : secret "maas-api-e2e-migrate-65fd3bf5-serving-cert" not found
Jul 14 15:47:43.243819 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.243803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-openshift-service-ca\") pod \"maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc\" (UID: \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:47:43.252922 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.252900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2wpq\" (UniqueName: \"kubernetes.io/projected/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-kube-api-access-s2wpq\") pod \"maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc\" (UID: \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:47:43.748155 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.748120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-maas-api-tls\") pod \"maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc\" (UID: \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:47:43.750718 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.750684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-maas-api-tls\") pod \"maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc\" (UID: \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:47:43.879413 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:43.879379    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:44.011878 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:44.011788    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:47:44.146417 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:44.146386    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"]
Jul 14 15:47:44.150268 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:47:44.150235    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ff61f1_75b5_4d66_b63a_dbc4bc73af26.slice/crio-b54febefa7cc4c690c0d354229a1507c7bda4483d894331395dcf19fecb26cde WatchSource:0}: Error finding container b54febefa7cc4c690c0d354229a1507c7bda4483d894331395dcf19fecb26cde: Status 404 returned error can't find the container with id b54febefa7cc4c690c0d354229a1507c7bda4483d894331395dcf19fecb26cde
Jul 14 15:47:44.252686 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:44.252658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc" event={"ID":"63ff61f1-75b5-4d66-b63a-dbc4bc73af26","Type":"ContainerStarted","Data":"b54febefa7cc4c690c0d354229a1507c7bda4483d894331395dcf19fecb26cde"}
Jul 14 15:47:44.252952 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:44.252938    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:44.254049 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:44.254030    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:47:44.595130 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:44.595053    2563 patch_prober.go:28] interesting pod/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 15:47:44.595278 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:44.595123    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj" podUID="d4001f1a-d506-4f09-ac97-bef509545c70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 15:47:45.259320 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:45.259289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc" event={"ID":"63ff61f1-75b5-4d66-b63a-dbc4bc73af26","Type":"ContainerStarted","Data":"155a146aa6d60f88b8f9e9a211b88a633dee285b47327fd8fb999f20e30d4ada"}
Jul 14 15:47:45.259711 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:45.259360    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:47:45.278662 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:45.278617    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc" podStartSLOduration=1.6224281029999998 podStartE2EDuration="2.278601983s" totalImagesPullingTime="656.17388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:47:45.276529889 +0000 UTC m=+1589.617795817" watchObservedRunningTime="2026-07-14 15:47:45.278601983 +0000 UTC m=+1589.619867886"
Jul 14 15:47:46.207542 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.207474    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:47:46.266325 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.266299    2563 generic.go:356] "Generic (PLEG): container finished" podID="d4001f1a-d506-4f09-ac97-bef509545c70" containerID="83b85e4ddb480a0629617042302b580974667d2ee55897a883250a160a8fd87d" exitCode=0
Jul 14 15:47:46.266835 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.266347    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"
Jul 14 15:47:46.266835 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.266387    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj" event={"ID":"d4001f1a-d506-4f09-ac97-bef509545c70","Type":"ContainerDied","Data":"83b85e4ddb480a0629617042302b580974667d2ee55897a883250a160a8fd87d"}
Jul 14 15:47:46.266835 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.266436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj" event={"ID":"d4001f1a-d506-4f09-ac97-bef509545c70","Type":"ContainerDied","Data":"e354d8d730da57dc86496500c8657761ef751bf86fde2fc30668f3b1478ef81e"}
Jul 14 15:47:46.266835 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.266465    2563 scope.go:117] "RemoveContainer" containerID="83b85e4ddb480a0629617042302b580974667d2ee55897a883250a160a8fd87d"
Jul 14 15:47:46.270539 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.270515    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-workload-socket\") pod \"d4001f1a-d506-4f09-ac97-bef509545c70\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") "
Jul 14 15:47:46.270695 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.270600    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4001f1a-d506-4f09-ac97-bef509545c70-istiod-ca-cert\") pod \"d4001f1a-d506-4f09-ac97-bef509545c70\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") "
Jul 14 15:47:46.270695 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.270635    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjrf6\" (UniqueName: \"kubernetes.io/projected/d4001f1a-d506-4f09-ac97-bef509545c70-kube-api-access-mjrf6\") pod \"d4001f1a-d506-4f09-ac97-bef509545c70\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") "
Jul 14 15:47:46.270695 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.270665    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-istio-data\") pod \"d4001f1a-d506-4f09-ac97-bef509545c70\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") "
Jul 14 15:47:46.270865 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.270802    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d4001f1a-d506-4f09-ac97-bef509545c70" (UID: "d4001f1a-d506-4f09-ac97-bef509545c70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:47:46.270986 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.270968    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-istio-envoy\") pod \"d4001f1a-d506-4f09-ac97-bef509545c70\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") "
Jul 14 15:47:46.271138 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.271116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4001f1a-d506-4f09-ac97-bef509545c70-istio-token\") pod \"d4001f1a-d506-4f09-ac97-bef509545c70\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") "
Jul 14 15:47:46.271280 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.271152    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4001f1a-d506-4f09-ac97-bef509545c70-istio-podinfo\") pod \"d4001f1a-d506-4f09-ac97-bef509545c70\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") "
Jul 14 15:47:46.271280 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.271211    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-workload-certs\") pod \"d4001f1a-d506-4f09-ac97-bef509545c70\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") "
Jul 14 15:47:46.271280 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.271253    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-credential-socket\") pod \"d4001f1a-d506-4f09-ac97-bef509545c70\" (UID: \"d4001f1a-d506-4f09-ac97-bef509545c70\") "
Jul 14 15:47:46.272312 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.272284    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-workload-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:47:46.272632 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.272604    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4001f1a-d506-4f09-ac97-bef509545c70-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d4001f1a-d506-4f09-ac97-bef509545c70" (UID: "d4001f1a-d506-4f09-ac97-bef509545c70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:47:46.272724 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.272690    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d4001f1a-d506-4f09-ac97-bef509545c70" (UID: "d4001f1a-d506-4f09-ac97-bef509545c70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:47:46.272791 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.272751    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-istio-data" (OuterVolumeSpecName: "istio-data") pod "d4001f1a-d506-4f09-ac97-bef509545c70" (UID: "d4001f1a-d506-4f09-ac97-bef509545c70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:47:46.273522 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.273476    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d4001f1a-d506-4f09-ac97-bef509545c70" (UID: "d4001f1a-d506-4f09-ac97-bef509545c70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:47:46.273842 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.273815    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d4001f1a-d506-4f09-ac97-bef509545c70" (UID: "d4001f1a-d506-4f09-ac97-bef509545c70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:47:46.276044 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.276020    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4001f1a-d506-4f09-ac97-bef509545c70-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d4001f1a-d506-4f09-ac97-bef509545c70" (UID: "d4001f1a-d506-4f09-ac97-bef509545c70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 15:47:46.284852 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.284813    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4001f1a-d506-4f09-ac97-bef509545c70-kube-api-access-mjrf6" (OuterVolumeSpecName: "kube-api-access-mjrf6") pod "d4001f1a-d506-4f09-ac97-bef509545c70" (UID: "d4001f1a-d506-4f09-ac97-bef509545c70"). InnerVolumeSpecName "kube-api-access-mjrf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:47:46.286444 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.286317    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4001f1a-d506-4f09-ac97-bef509545c70-istio-token" (OuterVolumeSpecName: "istio-token") pod "d4001f1a-d506-4f09-ac97-bef509545c70" (UID: "d4001f1a-d506-4f09-ac97-bef509545c70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:47:46.290232 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.290210    2563 scope.go:117] "RemoveContainer" containerID="83b85e4ddb480a0629617042302b580974667d2ee55897a883250a160a8fd87d"
Jul 14 15:47:46.290665 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:47:46.290641    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b85e4ddb480a0629617042302b580974667d2ee55897a883250a160a8fd87d\": container with ID starting with 83b85e4ddb480a0629617042302b580974667d2ee55897a883250a160a8fd87d not found: ID does not exist" containerID="83b85e4ddb480a0629617042302b580974667d2ee55897a883250a160a8fd87d"
Jul 14 15:47:46.290723 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.290678    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b85e4ddb480a0629617042302b580974667d2ee55897a883250a160a8fd87d"} err="failed to get container status \"83b85e4ddb480a0629617042302b580974667d2ee55897a883250a160a8fd87d\": rpc error: code = NotFound desc = could not find container \"83b85e4ddb480a0629617042302b580974667d2ee55897a883250a160a8fd87d\": container with ID starting with 83b85e4ddb480a0629617042302b580974667d2ee55897a883250a160a8fd87d not found: ID does not exist"
Jul 14 15:47:46.373965 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.373943    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-workload-certs\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:47:46.373965 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.373964    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-credential-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:47:46.374136 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.373975    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4001f1a-d506-4f09-ac97-bef509545c70-istiod-ca-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:47:46.374136 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.373984    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjrf6\" (UniqueName: \"kubernetes.io/projected/d4001f1a-d506-4f09-ac97-bef509545c70-kube-api-access-mjrf6\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:47:46.374136 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.373993    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-istio-data\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:47:46.374136 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.374001    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4001f1a-d506-4f09-ac97-bef509545c70-istio-envoy\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:47:46.374136 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.374008    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4001f1a-d506-4f09-ac97-bef509545c70-istio-token\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:47:46.374136 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.374016    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4001f1a-d506-4f09-ac97-bef509545c70-istio-podinfo\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:47:46.591926 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.591894    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"]
Jul 14 15:47:46.596560 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:46.596535    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-94af7c4d-openshift-default-584f7889ff-69wlj"]
Jul 14 15:47:47.503890 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:47.503856    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2"]
Jul 14 15:47:47.504308 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:47.504296    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4001f1a-d506-4f09-ac97-bef509545c70" containerName="istio-proxy"
Jul 14 15:47:47.504357 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:47.504310    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4001f1a-d506-4f09-ac97-bef509545c70" containerName="istio-proxy"
Jul 14 15:47:47.504399 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:47.504392    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4001f1a-d506-4f09-ac97-bef509545c70" containerName="istio-proxy"
Jul 14 15:47:47.508874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:47.508860    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2"
Jul 14 15:47:47.512155 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:47.512132    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\""
Jul 14 15:47:47.513991 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:47.513972    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2"]
Jul 14 15:47:47.587132 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:47.587045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15791585-05a5-4330-aa92-4cd408e72989-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2\" (UID: \"15791585-05a5-4330-aa92-4cd408e72989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2"
Jul 14 15:47:47.688576 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:47.688541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15791585-05a5-4330-aa92-4cd408e72989-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2\" (UID: \"15791585-05a5-4330-aa92-4cd408e72989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2"
Jul 14 15:47:47.689152 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:47.689133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15791585-05a5-4330-aa92-4cd408e72989-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2\" (UID: \"15791585-05a5-4330-aa92-4cd408e72989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2"
Jul 14 15:47:47.821420 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:47.821391    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2"
Jul 14 15:47:47.951357 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:47.951332    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2"]
Jul 14 15:47:47.952922 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:47:47.952897    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15791585_05a5_4330_aa92_4cd408e72989.slice/crio-e8fcbaef895785244200029daa5296070601413ba285fa6952064dfe2e6194bd WatchSource:0}: Error finding container e8fcbaef895785244200029daa5296070601413ba285fa6952064dfe2e6194bd: Status 404 returned error can't find the container with id e8fcbaef895785244200029daa5296070601413ba285fa6952064dfe2e6194bd
Jul 14 15:47:48.217691 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:48.217661    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4001f1a-d506-4f09-ac97-bef509545c70" path="/var/lib/kubelet/pods/d4001f1a-d506-4f09-ac97-bef509545c70/volumes"
Jul 14 15:47:48.279970 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:48.279938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2" event={"ID":"15791585-05a5-4330-aa92-4cd408e72989","Type":"ContainerStarted","Data":"d758a3a9af67e955aea7532ee3145931361e04ef4eff392af24a6e36eac8b6e6"}
Jul 14 15:47:48.280129 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:48.279998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2" event={"ID":"15791585-05a5-4330-aa92-4cd408e72989","Type":"ContainerStarted","Data":"e8fcbaef895785244200029daa5296070601413ba285fa6952064dfe2e6194bd"}
Jul 14 15:47:48.296443 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:48.296397    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2" podStartSLOduration=1.2963827110000001 podStartE2EDuration="1.296382711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:47:48.294725008 +0000 UTC m=+1592.635990915" watchObservedRunningTime="2026-07-14 15:47:48.296382711 +0000 UTC m=+1592.637648617"
Jul 14 15:47:49.286077 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:49.286042    2563 generic.go:356] "Generic (PLEG): container finished" podID="15791585-05a5-4330-aa92-4cd408e72989" containerID="d758a3a9af67e955aea7532ee3145931361e04ef4eff392af24a6e36eac8b6e6" exitCode=7
Jul 14 15:47:49.286494 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:49.286134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2" event={"ID":"15791585-05a5-4330-aa92-4cd408e72989","Type":"ContainerDied","Data":"d758a3a9af67e955aea7532ee3145931361e04ef4eff392af24a6e36eac8b6e6"}
Jul 14 15:47:49.286494 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:49.286299    2563 scope.go:117] "RemoveContainer" containerID="d758a3a9af67e955aea7532ee3145931361e04ef4eff392af24a6e36eac8b6e6"
Jul 14 15:47:50.292973 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:50.292881    2563 generic.go:356] "Generic (PLEG): container finished" podID="15791585-05a5-4330-aa92-4cd408e72989" containerID="b6dbc1543b0318449cd6a4e0c2a35f558b0469169eda7d300044262bae0feba3" exitCode=7
Jul 14 15:47:50.293388 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:50.292965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2" event={"ID":"15791585-05a5-4330-aa92-4cd408e72989","Type":"ContainerDied","Data":"b6dbc1543b0318449cd6a4e0c2a35f558b0469169eda7d300044262bae0feba3"}
Jul 14 15:47:50.293388 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:50.293018    2563 scope.go:117] "RemoveContainer" containerID="d758a3a9af67e955aea7532ee3145931361e04ef4eff392af24a6e36eac8b6e6"
Jul 14 15:47:50.293388 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:50.293133    2563 scope.go:117] "RemoveContainer" containerID="b6dbc1543b0318449cd6a4e0c2a35f558b0469169eda7d300044262bae0feba3"
Jul 14 15:47:50.293388 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:47:50.293379    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2_odh-ai-gateway-infra(15791585-05a5-4330-aa92-4cd408e72989)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2" podUID="15791585-05a5-4330-aa92-4cd408e72989"
Jul 14 15:47:51.272205 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:51.272174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:47:51.302686 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:51.302656    2563 scope.go:117] "RemoveContainer" containerID="b6dbc1543b0318449cd6a4e0c2a35f558b0469169eda7d300044262bae0feba3"
Jul 14 15:47:51.303046 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:47:51.302919    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2_odh-ai-gateway-infra(15791585-05a5-4330-aa92-4cd408e72989)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2" podUID="15791585-05a5-4330-aa92-4cd408e72989"
Jul 14 15:47:59.527213 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:59.527180    2563 patch_prober.go:28] interesting pod/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": read tcp 10.134.0.2:44096->10.134.0.48:15021: read: connection reset by peer" start-of-body=
Jul 14 15:47:59.527635 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:47:59.527247    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6" podUID="ef7abe83-94f1-41f5-ac58-e66792293425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": read tcp 10.134.0.2:44096->10.134.0.48:15021: read: connection reset by peer"
Jul 14 15:48:00.359269 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:00.359234    2563 generic.go:356] "Generic (PLEG): container finished" podID="ef7abe83-94f1-41f5-ac58-e66792293425" containerID="325a099d729794602d6e888487338e57eaccc43394b0856be5f7ad229ce7864b" exitCode=0
Jul 14 15:48:00.359449 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:00.359309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6" event={"ID":"ef7abe83-94f1-41f5-ac58-e66792293425","Type":"ContainerDied","Data":"325a099d729794602d6e888487338e57eaccc43394b0856be5f7ad229ce7864b"}
Jul 14 15:48:00.359792 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:00.359774    2563 scope.go:117] "RemoveContainer" containerID="325a099d729794602d6e888487338e57eaccc43394b0856be5f7ad229ce7864b"
Jul 14 15:48:00.360911 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:00.360879    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:48:00.360989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:00.360974    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:48:00.361068 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:00.361016    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:48:01.367404 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:01.367361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6" event={"ID":"ef7abe83-94f1-41f5-ac58-e66792293425","Type":"ContainerStarted","Data":"004c93171a391be005ff5959a315ba3693ab723a66d42eceb3f330ec2b9fa7e9"}
Jul 14 15:48:01.875094 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:01.875057    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:48:01.875317 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:01.875291    2563 patch_prober.go:28] interesting pod/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Jul 14 15:48:01.875370 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:01.875343    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6" podUID="ef7abe83-94f1-41f5-ac58-e66792293425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Jul 14 15:48:02.875707 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:02.875669    2563 patch_prober.go:28] interesting pod/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Jul 14 15:48:02.876187 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:02.875735    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6" podUID="ef7abe83-94f1-41f5-ac58-e66792293425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Jul 14 15:48:03.875526 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:03.875459    2563 patch_prober.go:28] interesting pod/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Jul 14 15:48:03.875713 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:03.875564    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6" podUID="ef7abe83-94f1-41f5-ac58-e66792293425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Jul 14 15:48:04.874926 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:04.874892    2563 patch_prober.go:28] interesting pod/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Jul 14 15:48:04.875345 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:04.874962    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6" podUID="ef7abe83-94f1-41f5-ac58-e66792293425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Jul 14 15:48:05.210566 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:05.210533    2563 scope.go:117] "RemoveContainer" containerID="b6dbc1543b0318449cd6a4e0c2a35f558b0469169eda7d300044262bae0feba3"
Jul 14 15:48:05.878039 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:05.878012    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:48:05.878403 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:05.878384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:48:05.879256 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:05.879241    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:48:06.396396 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:06.396369    2563 generic.go:356] "Generic (PLEG): container finished" podID="15791585-05a5-4330-aa92-4cd408e72989" containerID="dc1bc9b7fdd89347dc9de0f2f67fa0560051a10c3006182fdbb35616150560f9" exitCode=0
Jul 14 15:48:06.396578 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:06.396447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2" event={"ID":"15791585-05a5-4330-aa92-4cd408e72989","Type":"ContainerDied","Data":"dc1bc9b7fdd89347dc9de0f2f67fa0560051a10c3006182fdbb35616150560f9"}
Jul 14 15:48:06.396578 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:06.396524    2563 scope.go:117] "RemoveContainer" containerID="b6dbc1543b0318449cd6a4e0c2a35f558b0469169eda7d300044262bae0feba3"
Jul 14 15:48:07.434596 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:07.434562    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2"]
Jul 14 15:48:07.560051 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:07.560023    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2"
Jul 14 15:48:07.695635 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:07.695495    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15791585-05a5-4330-aa92-4cd408e72989-maas-api-service-ca\") pod \"15791585-05a5-4330-aa92-4cd408e72989\" (UID: \"15791585-05a5-4330-aa92-4cd408e72989\") "
Jul 14 15:48:07.696392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:07.696357    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15791585-05a5-4330-aa92-4cd408e72989-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "15791585-05a5-4330-aa92-4cd408e72989" (UID: "15791585-05a5-4330-aa92-4cd408e72989"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:48:07.796625 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:07.796591    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15791585-05a5-4330-aa92-4cd408e72989-maas-api-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:48:08.410629 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:08.410599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2" event={"ID":"15791585-05a5-4330-aa92-4cd408e72989","Type":"ContainerDied","Data":"e8fcbaef895785244200029daa5296070601413ba285fa6952064dfe2e6194bd"}
Jul 14 15:48:08.410629 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:08.410625    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2"
Jul 14 15:48:08.410844 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:08.410646    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8fcbaef895785244200029daa5296070601413ba285fa6952064dfe2e6194bd"
Jul 14 15:48:08.443467 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:08.443432    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2"]
Jul 14 15:48:08.449399 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:08.449370    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-2d4h2"]
Jul 14 15:48:10.216567 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:10.216534    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15791585-05a5-4330-aa92-4cd408e72989" path="/var/lib/kubelet/pods/15791585-05a5-4330-aa92-4cd408e72989/volumes"
Jul 14 15:48:47.555279 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.555249    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7"]
Jul 14 15:48:47.555744 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.555688    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15791585-05a5-4330-aa92-4cd408e72989" containerName="revoke-keys"
Jul 14 15:48:47.555744 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.555699    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15791585-05a5-4330-aa92-4cd408e72989" containerName="revoke-keys"
Jul 14 15:48:47.555744 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.555712    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15791585-05a5-4330-aa92-4cd408e72989" containerName="revoke-keys"
Jul 14 15:48:47.555744 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.555717    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15791585-05a5-4330-aa92-4cd408e72989" containerName="revoke-keys"
Jul 14 15:48:47.555744 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.555726    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15791585-05a5-4330-aa92-4cd408e72989" containerName="revoke-keys"
Jul 14 15:48:47.555744 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.555733    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15791585-05a5-4330-aa92-4cd408e72989" containerName="revoke-keys"
Jul 14 15:48:47.555935 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.555796    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="15791585-05a5-4330-aa92-4cd408e72989" containerName="revoke-keys"
Jul 14 15:48:47.555935 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.555803    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="15791585-05a5-4330-aa92-4cd408e72989" containerName="revoke-keys"
Jul 14 15:48:47.558860 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.558843    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7"
Jul 14 15:48:47.561614 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.561590    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\""
Jul 14 15:48:47.568534 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.568493    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7"]
Jul 14 15:48:47.569518 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.569473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b8feca-eb12-4b86-9634-280d60af3171-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7\" (UID: \"06b8feca-eb12-4b86-9634-280d60af3171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7"
Jul 14 15:48:47.631898 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.631863    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"]
Jul 14 15:48:47.632088 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.632051    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6" podUID="ef7abe83-94f1-41f5-ac58-e66792293425" containerName="istio-proxy" containerID="cri-o://004c93171a391be005ff5959a315ba3693ab723a66d42eceb3f330ec2b9fa7e9" gracePeriod=30
Jul 14 15:48:47.670619 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.670588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b8feca-eb12-4b86-9634-280d60af3171-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7\" (UID: \"06b8feca-eb12-4b86-9634-280d60af3171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7"
Jul 14 15:48:47.671174 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.671156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b8feca-eb12-4b86-9634-280d60af3171-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7\" (UID: \"06b8feca-eb12-4b86-9634-280d60af3171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7"
Jul 14 15:48:47.869973 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:47.869888    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7"
Jul 14 15:48:48.001274 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:48.001249    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7"]
Jul 14 15:48:48.002819 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:48:48.002794    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b8feca_eb12_4b86_9634_280d60af3171.slice/crio-26e1ef0ae040f93337a28dad9ac1daa5b28ca15273e49f53b967d445aa0f8380 WatchSource:0}: Error finding container 26e1ef0ae040f93337a28dad9ac1daa5b28ca15273e49f53b967d445aa0f8380: Status 404 returned error can't find the container with id 26e1ef0ae040f93337a28dad9ac1daa5b28ca15273e49f53b967d445aa0f8380
Jul 14 15:48:48.620406 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:48.620319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7" event={"ID":"06b8feca-eb12-4b86-9634-280d60af3171","Type":"ContainerStarted","Data":"baf048dcb6542ef130e590e3c332362f45340282e319021bdf12a95012ff0d76"}
Jul 14 15:48:48.620406 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:48.620373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7" event={"ID":"06b8feca-eb12-4b86-9634-280d60af3171","Type":"ContainerStarted","Data":"26e1ef0ae040f93337a28dad9ac1daa5b28ca15273e49f53b967d445aa0f8380"}
Jul 14 15:48:48.642722 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:48.642678    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7" podStartSLOduration=1.642663429 podStartE2EDuration="1.642663429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:48:48.641034786 +0000 UTC m=+1652.982300726" watchObservedRunningTime="2026-07-14 15:48:48.642663429 +0000 UTC m=+1652.983929335"
Jul 14 15:48:49.627856 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:49.627820    2563 generic.go:356] "Generic (PLEG): container finished" podID="06b8feca-eb12-4b86-9634-280d60af3171" containerID="baf048dcb6542ef130e590e3c332362f45340282e319021bdf12a95012ff0d76" exitCode=0
Jul 14 15:48:49.628223 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:49.627906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7" event={"ID":"06b8feca-eb12-4b86-9634-280d60af3171","Type":"ContainerDied","Data":"baf048dcb6542ef130e590e3c332362f45340282e319021bdf12a95012ff0d76"}
Jul 14 15:48:50.773900 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:50.773874    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7"
Jul 14 15:48:50.799153 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:50.799124    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b8feca-eb12-4b86-9634-280d60af3171-maas-api-service-ca\") pod \"06b8feca-eb12-4b86-9634-280d60af3171\" (UID: \"06b8feca-eb12-4b86-9634-280d60af3171\") "
Jul 14 15:48:50.799559 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:50.799532    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b8feca-eb12-4b86-9634-280d60af3171-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "06b8feca-eb12-4b86-9634-280d60af3171" (UID: "06b8feca-eb12-4b86-9634-280d60af3171"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:48:50.880933 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:50.880896    2563 patch_prober.go:28] interesting pod/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 15:48:50.881098 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:50.880967    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6" podUID="ef7abe83-94f1-41f5-ac58-e66792293425" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 15:48:50.900394 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:50.900327    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b8feca-eb12-4b86-9634-280d60af3171-maas-api-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:48:51.639793 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:51.639763    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7"
Jul 14 15:48:51.639963 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:51.639798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7" event={"ID":"06b8feca-eb12-4b86-9634-280d60af3171","Type":"ContainerDied","Data":"26e1ef0ae040f93337a28dad9ac1daa5b28ca15273e49f53b967d445aa0f8380"}
Jul 14 15:48:51.639963 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:51.639843    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26e1ef0ae040f93337a28dad9ac1daa5b28ca15273e49f53b967d445aa0f8380"
Jul 14 15:48:52.891618 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.891556    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:48:52.921084 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921048    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef7abe83-94f1-41f5-ac58-e66792293425-istio-token\") pod \"ef7abe83-94f1-41f5-ac58-e66792293425\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") "
Jul 14 15:48:52.921258 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921102    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-workload-certs\") pod \"ef7abe83-94f1-41f5-ac58-e66792293425\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") "
Jul 14 15:48:52.921258 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921165    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-workload-socket\") pod \"ef7abe83-94f1-41f5-ac58-e66792293425\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") "
Jul 14 15:48:52.921258 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921193    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-credential-socket\") pod \"ef7abe83-94f1-41f5-ac58-e66792293425\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") "
Jul 14 15:48:52.921258 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921235    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smlh7\" (UniqueName: \"kubernetes.io/projected/ef7abe83-94f1-41f5-ac58-e66792293425-kube-api-access-smlh7\") pod \"ef7abe83-94f1-41f5-ac58-e66792293425\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") "
Jul 14 15:48:52.921258 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921258    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef7abe83-94f1-41f5-ac58-e66792293425-istio-podinfo\") pod \"ef7abe83-94f1-41f5-ac58-e66792293425\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") "
Jul 14 15:48:52.921629 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921327    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef7abe83-94f1-41f5-ac58-e66792293425-istiod-ca-cert\") pod \"ef7abe83-94f1-41f5-ac58-e66792293425\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") "
Jul 14 15:48:52.921629 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921400    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-istio-envoy\") pod \"ef7abe83-94f1-41f5-ac58-e66792293425\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") "
Jul 14 15:48:52.921629 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921465    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-istio-data\") pod \"ef7abe83-94f1-41f5-ac58-e66792293425\" (UID: \"ef7abe83-94f1-41f5-ac58-e66792293425\") "
Jul 14 15:48:52.921629 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921467    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ef7abe83-94f1-41f5-ac58-e66792293425" (UID: "ef7abe83-94f1-41f5-ac58-e66792293425"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:48:52.921629 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921485    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ef7abe83-94f1-41f5-ac58-e66792293425" (UID: "ef7abe83-94f1-41f5-ac58-e66792293425"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:48:52.921629 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921525    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ef7abe83-94f1-41f5-ac58-e66792293425" (UID: "ef7abe83-94f1-41f5-ac58-e66792293425"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:48:52.921957 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921809    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7abe83-94f1-41f5-ac58-e66792293425-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ef7abe83-94f1-41f5-ac58-e66792293425" (UID: "ef7abe83-94f1-41f5-ac58-e66792293425"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:48:52.921957 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921834    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-workload-certs\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:48:52.921957 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921855    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-workload-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:48:52.921957 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.921869    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-credential-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:48:52.922570 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.922544    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-istio-data" (OuterVolumeSpecName: "istio-data") pod "ef7abe83-94f1-41f5-ac58-e66792293425" (UID: "ef7abe83-94f1-41f5-ac58-e66792293425"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:48:52.923719 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.923693    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef7abe83-94f1-41f5-ac58-e66792293425-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ef7abe83-94f1-41f5-ac58-e66792293425" (UID: "ef7abe83-94f1-41f5-ac58-e66792293425"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 15:48:52.923849 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.923822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7abe83-94f1-41f5-ac58-e66792293425-istio-token" (OuterVolumeSpecName: "istio-token") pod "ef7abe83-94f1-41f5-ac58-e66792293425" (UID: "ef7abe83-94f1-41f5-ac58-e66792293425"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:48:52.924115 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.924094    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ef7abe83-94f1-41f5-ac58-e66792293425" (UID: "ef7abe83-94f1-41f5-ac58-e66792293425"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:48:52.924179 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:52.924094    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7abe83-94f1-41f5-ac58-e66792293425-kube-api-access-smlh7" (OuterVolumeSpecName: "kube-api-access-smlh7") pod "ef7abe83-94f1-41f5-ac58-e66792293425" (UID: "ef7abe83-94f1-41f5-ac58-e66792293425"). InnerVolumeSpecName "kube-api-access-smlh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:48:53.022663 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.022625    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-istio-data\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:48:53.022663 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.022657    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef7abe83-94f1-41f5-ac58-e66792293425-istio-token\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:48:53.022663 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.022670    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smlh7\" (UniqueName: \"kubernetes.io/projected/ef7abe83-94f1-41f5-ac58-e66792293425-kube-api-access-smlh7\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:48:53.022907 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.022683    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef7abe83-94f1-41f5-ac58-e66792293425-istio-podinfo\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:48:53.022907 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.022694    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef7abe83-94f1-41f5-ac58-e66792293425-istiod-ca-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:48:53.022907 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.022706    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef7abe83-94f1-41f5-ac58-e66792293425-istio-envoy\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:48:53.552213 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.552180    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"]
Jul 14 15:48:53.552479 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.552432    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc" podUID="63ff61f1-75b5-4d66-b63a-dbc4bc73af26" containerName="maas-api" containerID="cri-o://155a146aa6d60f88b8f9e9a211b88a633dee285b47327fd8fb999f20e30d4ada" gracePeriod=30
Jul 14 15:48:53.653277 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.653242    2563 generic.go:356] "Generic (PLEG): container finished" podID="ef7abe83-94f1-41f5-ac58-e66792293425" containerID="004c93171a391be005ff5959a315ba3693ab723a66d42eceb3f330ec2b9fa7e9" exitCode=0
Jul 14 15:48:53.653423 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.653313    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"
Jul 14 15:48:53.653423 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.653302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6" event={"ID":"ef7abe83-94f1-41f5-ac58-e66792293425","Type":"ContainerDied","Data":"004c93171a391be005ff5959a315ba3693ab723a66d42eceb3f330ec2b9fa7e9"}
Jul 14 15:48:53.653528 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.653444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6" event={"ID":"ef7abe83-94f1-41f5-ac58-e66792293425","Type":"ContainerDied","Data":"b194ba0cb5e2c1f5643afda484aac6fdb3a74ff5cd5d495b2bf2614ab454daa2"}
Jul 14 15:48:53.653528 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.653468    2563 scope.go:117] "RemoveContainer" containerID="004c93171a391be005ff5959a315ba3693ab723a66d42eceb3f330ec2b9fa7e9"
Jul 14 15:48:53.665560 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.665539    2563 scope.go:117] "RemoveContainer" containerID="325a099d729794602d6e888487338e57eaccc43394b0856be5f7ad229ce7864b"
Jul 14 15:48:53.674898 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.674871    2563 scope.go:117] "RemoveContainer" containerID="004c93171a391be005ff5959a315ba3693ab723a66d42eceb3f330ec2b9fa7e9"
Jul 14 15:48:53.675342 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:48:53.675313    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004c93171a391be005ff5959a315ba3693ab723a66d42eceb3f330ec2b9fa7e9\": container with ID starting with 004c93171a391be005ff5959a315ba3693ab723a66d42eceb3f330ec2b9fa7e9 not found: ID does not exist" containerID="004c93171a391be005ff5959a315ba3693ab723a66d42eceb3f330ec2b9fa7e9"
Jul 14 15:48:53.675454 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.675351    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004c93171a391be005ff5959a315ba3693ab723a66d42eceb3f330ec2b9fa7e9"} err="failed to get container status \"004c93171a391be005ff5959a315ba3693ab723a66d42eceb3f330ec2b9fa7e9\": rpc error: code = NotFound desc = could not find container \"004c93171a391be005ff5959a315ba3693ab723a66d42eceb3f330ec2b9fa7e9\": container with ID starting with 004c93171a391be005ff5959a315ba3693ab723a66d42eceb3f330ec2b9fa7e9 not found: ID does not exist"
Jul 14 15:48:53.675454 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.675371    2563 scope.go:117] "RemoveContainer" containerID="325a099d729794602d6e888487338e57eaccc43394b0856be5f7ad229ce7864b"
Jul 14 15:48:53.675719 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:48:53.675698    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325a099d729794602d6e888487338e57eaccc43394b0856be5f7ad229ce7864b\": container with ID starting with 325a099d729794602d6e888487338e57eaccc43394b0856be5f7ad229ce7864b not found: ID does not exist" containerID="325a099d729794602d6e888487338e57eaccc43394b0856be5f7ad229ce7864b"
Jul 14 15:48:53.675782 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.675728    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325a099d729794602d6e888487338e57eaccc43394b0856be5f7ad229ce7864b"} err="failed to get container status \"325a099d729794602d6e888487338e57eaccc43394b0856be5f7ad229ce7864b\": rpc error: code = NotFound desc = could not find container \"325a099d729794602d6e888487338e57eaccc43394b0856be5f7ad229ce7864b\": container with ID starting with 325a099d729794602d6e888487338e57eaccc43394b0856be5f7ad229ce7864b not found: ID does not exist"
Jul 14 15:48:53.679843 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.679643    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"]
Jul 14 15:48:53.682115 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.682095    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-65fd3bf5-gw-openshift-default-dff6c56bc-hzvg6"]
Jul 14 15:48:53.820176 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.820111    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:48:53.931464 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.931431    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2wpq\" (UniqueName: \"kubernetes.io/projected/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-kube-api-access-s2wpq\") pod \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\" (UID: \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\") "
Jul 14 15:48:53.931941 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.931476    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-openshift-service-ca\") pod \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\" (UID: \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\") "
Jul 14 15:48:53.931941 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.931560    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-maas-api-tls\") pod \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\" (UID: \"63ff61f1-75b5-4d66-b63a-dbc4bc73af26\") "
Jul 14 15:48:53.931941 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.931887    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63ff61f1-75b5-4d66-b63a-dbc4bc73af26" (UID: "63ff61f1-75b5-4d66-b63a-dbc4bc73af26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:48:53.933796 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.933772    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-kube-api-access-s2wpq" (OuterVolumeSpecName: "kube-api-access-s2wpq") pod "63ff61f1-75b5-4d66-b63a-dbc4bc73af26" (UID: "63ff61f1-75b5-4d66-b63a-dbc4bc73af26"). InnerVolumeSpecName "kube-api-access-s2wpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:48:53.934166 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:53.934147    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63ff61f1-75b5-4d66-b63a-dbc4bc73af26" (UID: "63ff61f1-75b5-4d66-b63a-dbc4bc73af26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:48:54.032800 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:54.032763    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2wpq\" (UniqueName: \"kubernetes.io/projected/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-kube-api-access-s2wpq\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:48:54.032800 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:54.032797    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-openshift-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:48:54.033013 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:54.032814    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63ff61f1-75b5-4d66-b63a-dbc4bc73af26-maas-api-tls\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:48:54.217040 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:54.217009    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef7abe83-94f1-41f5-ac58-e66792293425" path="/var/lib/kubelet/pods/ef7abe83-94f1-41f5-ac58-e66792293425/volumes"
Jul 14 15:48:54.660765 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:54.660731    2563 generic.go:356] "Generic (PLEG): container finished" podID="63ff61f1-75b5-4d66-b63a-dbc4bc73af26" containerID="155a146aa6d60f88b8f9e9a211b88a633dee285b47327fd8fb999f20e30d4ada" exitCode=0
Jul 14 15:48:54.660912 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:54.660777    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"
Jul 14 15:48:54.660912 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:54.660775    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc" event={"ID":"63ff61f1-75b5-4d66-b63a-dbc4bc73af26","Type":"ContainerDied","Data":"155a146aa6d60f88b8f9e9a211b88a633dee285b47327fd8fb999f20e30d4ada"}
Jul 14 15:48:54.660912 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:54.660898    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc" event={"ID":"63ff61f1-75b5-4d66-b63a-dbc4bc73af26","Type":"ContainerDied","Data":"b54febefa7cc4c690c0d354229a1507c7bda4483d894331395dcf19fecb26cde"}
Jul 14 15:48:54.661076 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:54.660927    2563 scope.go:117] "RemoveContainer" containerID="155a146aa6d60f88b8f9e9a211b88a633dee285b47327fd8fb999f20e30d4ada"
Jul 14 15:48:54.669821 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:54.669805    2563 scope.go:117] "RemoveContainer" containerID="155a146aa6d60f88b8f9e9a211b88a633dee285b47327fd8fb999f20e30d4ada"
Jul 14 15:48:54.670103 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:48:54.670074    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"155a146aa6d60f88b8f9e9a211b88a633dee285b47327fd8fb999f20e30d4ada\": container with ID starting with 155a146aa6d60f88b8f9e9a211b88a633dee285b47327fd8fb999f20e30d4ada not found: ID does not exist" containerID="155a146aa6d60f88b8f9e9a211b88a633dee285b47327fd8fb999f20e30d4ada"
Jul 14 15:48:54.670184 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:54.670110    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155a146aa6d60f88b8f9e9a211b88a633dee285b47327fd8fb999f20e30d4ada"} err="failed to get container status \"155a146aa6d60f88b8f9e9a211b88a633dee285b47327fd8fb999f20e30d4ada\": rpc error: code = NotFound desc = could not find container \"155a146aa6d60f88b8f9e9a211b88a633dee285b47327fd8fb999f20e30d4ada\": container with ID starting with 155a146aa6d60f88b8f9e9a211b88a633dee285b47327fd8fb999f20e30d4ada not found: ID does not exist"
Jul 14 15:48:54.681200 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:54.681173    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"]
Jul 14 15:48:54.683177 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:54.683152    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-65fd3bf5-67cdd4d544-5x6cc"]
Jul 14 15:48:56.217146 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:56.217113    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63ff61f1-75b5-4d66-b63a-dbc4bc73af26" path="/var/lib/kubelet/pods/63ff61f1-75b5-4d66-b63a-dbc4bc73af26/volumes"
Jul 14 15:48:59.805738 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.805703    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"]
Jul 14 15:48:59.806347 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.806329    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06b8feca-eb12-4b86-9634-280d60af3171" containerName="revoke-keys"
Jul 14 15:48:59.806406 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.806350    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="06b8feca-eb12-4b86-9634-280d60af3171" containerName="revoke-keys"
Jul 14 15:48:59.806406 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.806368    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7abe83-94f1-41f5-ac58-e66792293425" containerName="istio-proxy"
Jul 14 15:48:59.806406 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.806374    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7abe83-94f1-41f5-ac58-e66792293425" containerName="istio-proxy"
Jul 14 15:48:59.806406 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.806388    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63ff61f1-75b5-4d66-b63a-dbc4bc73af26" containerName="maas-api"
Jul 14 15:48:59.806406 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.806394    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="63ff61f1-75b5-4d66-b63a-dbc4bc73af26" containerName="maas-api"
Jul 14 15:48:59.806406 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.806405    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7abe83-94f1-41f5-ac58-e66792293425" containerName="istio-proxy"
Jul 14 15:48:59.806632 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.806410    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7abe83-94f1-41f5-ac58-e66792293425" containerName="istio-proxy"
Jul 14 15:48:59.806632 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.806489    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="63ff61f1-75b5-4d66-b63a-dbc4bc73af26" containerName="maas-api"
Jul 14 15:48:59.806632 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.806497    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="06b8feca-eb12-4b86-9634-280d60af3171" containerName="revoke-keys"
Jul 14 15:48:59.806632 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.806530    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7abe83-94f1-41f5-ac58-e66792293425" containerName="istio-proxy"
Jul 14 15:48:59.806632 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.806537    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7abe83-94f1-41f5-ac58-e66792293425" containerName="istio-proxy"
Jul 14 15:48:59.806632 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.806543    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="15791585-05a5-4330-aa92-4cd408e72989" containerName="revoke-keys"
Jul 14 15:48:59.811004 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.810985    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.814033 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.814010    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cd6a0c0f-openshift-default-dockercfg-fwsc7\""
Jul 14 15:48:59.814179 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.814036    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 15:48:59.814285 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.814059    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 15:48:59.814408 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.814270    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 15:48:59.822715 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.822692    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"]
Jul 14 15:48:59.884553 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.884488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-istio-data\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.884753 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.884569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-credential-socket\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.884753 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.884594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-istio-envoy\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.884753 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.884613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46967238-4e63-4fc8-be18-74d1a288c240-istio-token\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.884753 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.884695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkck\" (UniqueName: \"kubernetes.io/projected/46967238-4e63-4fc8-be18-74d1a288c240-kube-api-access-zpkck\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.884914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.884768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-workload-certs\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.884914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.884831    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-workload-socket\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.884914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.884852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46967238-4e63-4fc8-be18-74d1a288c240-istio-podinfo\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.884914 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.884886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46967238-4e63-4fc8-be18-74d1a288c240-istiod-ca-cert\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.985972 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.985940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46967238-4e63-4fc8-be18-74d1a288c240-istio-podinfo\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.986188 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.985992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46967238-4e63-4fc8-be18-74d1a288c240-istiod-ca-cert\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.986188 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.986016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-istio-data\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.986188 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.986051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-credential-socket\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.986188 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.986072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-istio-envoy\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.986188 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.986091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46967238-4e63-4fc8-be18-74d1a288c240-istio-token\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.986188 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.986138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkck\" (UniqueName: \"kubernetes.io/projected/46967238-4e63-4fc8-be18-74d1a288c240-kube-api-access-zpkck\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.986407 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.986264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-workload-certs\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.986407 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.986349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-workload-socket\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.986764 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.986734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-credential-socket\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.986764 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.986746    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46967238-4e63-4fc8-be18-74d1a288c240-istiod-ca-cert\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.986969 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.986834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-istio-data\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.986969 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.986880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-workload-socket\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.986969 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.986869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-workload-certs\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.988797 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.988780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-istio-envoy\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.989003 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.988986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46967238-4e63-4fc8-be18-74d1a288c240-istio-podinfo\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.997189 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.997163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46967238-4e63-4fc8-be18-74d1a288c240-istio-token\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:48:59.997582 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:48:59.997555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkck\" (UniqueName: \"kubernetes.io/projected/46967238-4e63-4fc8-be18-74d1a288c240-kube-api-access-zpkck\") pod \"e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") " pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:49:00.123757 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:00.123656    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:49:00.268772 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:00.268744    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"]
Jul 14 15:49:00.270574 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:49:00.270545    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46967238_4e63_4fc8_be18_74d1a288c240.slice/crio-f21a6532c967681fd8196d35d1cb1f86f2db70b9f36e5b095073403886120c0b WatchSource:0}: Error finding container f21a6532c967681fd8196d35d1cb1f86f2db70b9f36e5b095073403886120c0b: Status 404 returned error can't find the container with id f21a6532c967681fd8196d35d1cb1f86f2db70b9f36e5b095073403886120c0b
Jul 14 15:49:00.272712 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:00.272680    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:49:00.272776 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:00.272750    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:49:00.272816 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:00.272779    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:49:00.695031 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:00.694997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc" event={"ID":"46967238-4e63-4fc8-be18-74d1a288c240","Type":"ContainerStarted","Data":"d471a4a05f405da9b00f0e625376eea399442567c9a39bc49b5bd532152afe5f"}
Jul 14 15:49:00.695224 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:00.695051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc" event={"ID":"46967238-4e63-4fc8-be18-74d1a288c240","Type":"ContainerStarted","Data":"f21a6532c967681fd8196d35d1cb1f86f2db70b9f36e5b095073403886120c0b"}
Jul 14 15:49:00.718903 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:00.718841    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc" podStartSLOduration=1.718820534 podStartE2EDuration="1.718820534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:49:00.715922831 +0000 UTC m=+1665.057188756" watchObservedRunningTime="2026-07-14 15:49:00.718820534 +0000 UTC m=+1665.060086442"
Jul 14 15:49:01.123908 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.123804    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:49:01.130162 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.130132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:49:01.204661 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.204615    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"]
Jul 14 15:49:01.208682 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.208658    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:49:01.212952 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.212924    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cd6a0c0f-serving-cert\""
Jul 14 15:49:01.213120 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.213094    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 15:49:01.214006 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.213983    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npnj7\""
Jul 14 15:49:01.214215 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.214202    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 15:49:01.226851 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.226821    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"]
Jul 14 15:49:01.299690 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.299650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c78286f9-a66f-43b6-9932-c4396d7db0ca-maas-api-tls\") pod \"maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2\" (UID: \"c78286f9-a66f-43b6-9932-c4396d7db0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:49:01.299908 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.299793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78286f9-a66f-43b6-9932-c4396d7db0ca-openshift-service-ca\") pod \"maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2\" (UID: \"c78286f9-a66f-43b6-9932-c4396d7db0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:49:01.299908 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.299878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2wxg\" (UniqueName: \"kubernetes.io/projected/c78286f9-a66f-43b6-9932-c4396d7db0ca-kube-api-access-q2wxg\") pod \"maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2\" (UID: \"c78286f9-a66f-43b6-9932-c4396d7db0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:49:01.401344 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.401307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c78286f9-a66f-43b6-9932-c4396d7db0ca-maas-api-tls\") pod \"maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2\" (UID: \"c78286f9-a66f-43b6-9932-c4396d7db0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:49:01.401574 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.401375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78286f9-a66f-43b6-9932-c4396d7db0ca-openshift-service-ca\") pod \"maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2\" (UID: \"c78286f9-a66f-43b6-9932-c4396d7db0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:49:01.401574 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.401403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2wxg\" (UniqueName: \"kubernetes.io/projected/c78286f9-a66f-43b6-9932-c4396d7db0ca-kube-api-access-q2wxg\") pod \"maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2\" (UID: \"c78286f9-a66f-43b6-9932-c4396d7db0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:49:01.401574 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:49:01.401468    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-serving-cert: secret "maas-api-e2e-ait-cd6a0c0f-serving-cert" not found
Jul 14 15:49:01.401704 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:49:01.401592    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c78286f9-a66f-43b6-9932-c4396d7db0ca-maas-api-tls podName:c78286f9-a66f-43b6-9932-c4396d7db0ca nodeName:}" failed. No retries permitted until 2026-07-14 15:49:01.901570403 +0000 UTC m=+1666.242836300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c78286f9-a66f-43b6-9932-c4396d7db0ca-maas-api-tls") pod "maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2" (UID: "c78286f9-a66f-43b6-9932-c4396d7db0ca") : secret "maas-api-e2e-ait-cd6a0c0f-serving-cert" not found
Jul 14 15:49:01.402082 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.402062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78286f9-a66f-43b6-9932-c4396d7db0ca-openshift-service-ca\") pod \"maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2\" (UID: \"c78286f9-a66f-43b6-9932-c4396d7db0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:49:01.412088 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.412054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2wxg\" (UniqueName: \"kubernetes.io/projected/c78286f9-a66f-43b6-9932-c4396d7db0ca-kube-api-access-q2wxg\") pod \"maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2\" (UID: \"c78286f9-a66f-43b6-9932-c4396d7db0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:49:01.700828 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.700738    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:49:01.701861 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.701841    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:49:01.907200 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.907155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c78286f9-a66f-43b6-9932-c4396d7db0ca-maas-api-tls\") pod \"maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2\" (UID: \"c78286f9-a66f-43b6-9932-c4396d7db0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:49:01.909813 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:01.909786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c78286f9-a66f-43b6-9932-c4396d7db0ca-maas-api-tls\") pod \"maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2\" (UID: \"c78286f9-a66f-43b6-9932-c4396d7db0ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:49:02.121959 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:02.121852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:49:02.264660 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:02.264636    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"]
Jul 14 15:49:02.266602 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:49:02.266573    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78286f9_a66f_43b6_9932_c4396d7db0ca.slice/crio-f74fd1c512b0aa2fcf806ed9e35d20910c9ec59ec3acadcfbb4f8e84da68a06a WatchSource:0}: Error finding container f74fd1c512b0aa2fcf806ed9e35d20910c9ec59ec3acadcfbb4f8e84da68a06a: Status 404 returned error can't find the container with id f74fd1c512b0aa2fcf806ed9e35d20910c9ec59ec3acadcfbb4f8e84da68a06a
Jul 14 15:49:02.707728 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:02.707694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2" event={"ID":"c78286f9-a66f-43b6-9932-c4396d7db0ca","Type":"ContainerStarted","Data":"f74fd1c512b0aa2fcf806ed9e35d20910c9ec59ec3acadcfbb4f8e84da68a06a"}
Jul 14 15:49:03.714917 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:03.714880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2" event={"ID":"c78286f9-a66f-43b6-9932-c4396d7db0ca","Type":"ContainerStarted","Data":"5dc4c73c92cd80db944f76244eb15d639069cbe82428f00fd66ebe7832ab6df6"}
Jul 14 15:49:03.715360 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:03.714966    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:49:03.732802 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:03.732756    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2" podStartSLOduration=2.179271161 podStartE2EDuration="2.732739744s" totalImagesPullingTime="553.468583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:49:03.731782987 +0000 UTC m=+1668.073048921" watchObservedRunningTime="2026-07-14 15:49:03.732739744 +0000 UTC m=+1668.074005708"
Jul 14 15:49:06.361541 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:06.361487    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8"]
Jul 14 15:49:06.365563 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:06.365543    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8"
Jul 14 15:49:06.368198 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:06.368174    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\""
Jul 14 15:49:06.371937 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:06.371914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8"]
Jul 14 15:49:06.456995 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:06.456953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa63db74-4edb-4a3f-9023-63ecb6009dc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8\" (UID: \"fa63db74-4edb-4a3f-9023-63ecb6009dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8"
Jul 14 15:49:06.557594 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:06.557564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa63db74-4edb-4a3f-9023-63ecb6009dc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8\" (UID: \"fa63db74-4edb-4a3f-9023-63ecb6009dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8"
Jul 14 15:49:06.558330 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:06.558305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa63db74-4edb-4a3f-9023-63ecb6009dc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8\" (UID: \"fa63db74-4edb-4a3f-9023-63ecb6009dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8"
Jul 14 15:49:06.677753 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:06.677719    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8"
Jul 14 15:49:06.849169 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:06.849141    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8"]
Jul 14 15:49:06.850829 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:49:06.850791    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa63db74_4edb_4a3f_9023_63ecb6009dc2.slice/crio-7779444574bc5bfb560853368fbb871f3d956688e506bc4683d00ac90539315b WatchSource:0}: Error finding container 7779444574bc5bfb560853368fbb871f3d956688e506bc4683d00ac90539315b: Status 404 returned error can't find the container with id 7779444574bc5bfb560853368fbb871f3d956688e506bc4683d00ac90539315b
Jul 14 15:49:07.749384 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:07.749350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8" event={"ID":"fa63db74-4edb-4a3f-9023-63ecb6009dc2","Type":"ContainerStarted","Data":"fbd61705900c9f1cd502b22529999eda3d053d764fe1805a2b8272cfbe544cbd"}
Jul 14 15:49:07.749896 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:07.749418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8" event={"ID":"fa63db74-4edb-4a3f-9023-63ecb6009dc2","Type":"ContainerStarted","Data":"7779444574bc5bfb560853368fbb871f3d956688e506bc4683d00ac90539315b"}
Jul 14 15:49:07.766382 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:07.766331    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8" podStartSLOduration=1.7663168200000001 podStartE2EDuration="1.76631682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:49:07.76455761 +0000 UTC m=+1672.105823526" watchObservedRunningTime="2026-07-14 15:49:07.76631682 +0000 UTC m=+1672.107582720"
Jul 14 15:49:08.755859 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:08.755819    2563 generic.go:356] "Generic (PLEG): container finished" podID="fa63db74-4edb-4a3f-9023-63ecb6009dc2" containerID="fbd61705900c9f1cd502b22529999eda3d053d764fe1805a2b8272cfbe544cbd" exitCode=7
Jul 14 15:49:08.756260 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:08.755870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8" event={"ID":"fa63db74-4edb-4a3f-9023-63ecb6009dc2","Type":"ContainerDied","Data":"fbd61705900c9f1cd502b22529999eda3d053d764fe1805a2b8272cfbe544cbd"}
Jul 14 15:49:08.756260 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:08.756088    2563 scope.go:117] "RemoveContainer" containerID="fbd61705900c9f1cd502b22529999eda3d053d764fe1805a2b8272cfbe544cbd"
Jul 14 15:49:09.732785 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:09.732750    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:49:09.764403 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:09.764362    2563 generic.go:356] "Generic (PLEG): container finished" podID="fa63db74-4edb-4a3f-9023-63ecb6009dc2" containerID="2bdbe1de3d2b773bf4a5838f5cdcb692e9d60c7bafa1839d04869f2a7ee96d4d" exitCode=7
Jul 14 15:49:09.764861 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:09.764489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8" event={"ID":"fa63db74-4edb-4a3f-9023-63ecb6009dc2","Type":"ContainerDied","Data":"2bdbe1de3d2b773bf4a5838f5cdcb692e9d60c7bafa1839d04869f2a7ee96d4d"}
Jul 14 15:49:09.764861 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:09.764582    2563 scope.go:117] "RemoveContainer" containerID="fbd61705900c9f1cd502b22529999eda3d053d764fe1805a2b8272cfbe544cbd"
Jul 14 15:49:09.764861 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:09.764760    2563 scope.go:117] "RemoveContainer" containerID="2bdbe1de3d2b773bf4a5838f5cdcb692e9d60c7bafa1839d04869f2a7ee96d4d"
Jul 14 15:49:09.765028 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:49:09.764992    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8_odh-ai-gateway-infra(fa63db74-4edb-4a3f-9023-63ecb6009dc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8" podUID="fa63db74-4edb-4a3f-9023-63ecb6009dc2"
Jul 14 15:49:10.772156 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:10.772128    2563 scope.go:117] "RemoveContainer" containerID="2bdbe1de3d2b773bf4a5838f5cdcb692e9d60c7bafa1839d04869f2a7ee96d4d"
Jul 14 15:49:10.772552 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:49:10.772324    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8_odh-ai-gateway-infra(fa63db74-4edb-4a3f-9023-63ecb6009dc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8" podUID="fa63db74-4edb-4a3f-9023-63ecb6009dc2"
Jul 14 15:49:13.790852 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:13.790814    2563 generic.go:356] "Generic (PLEG): container finished" podID="46967238-4e63-4fc8-be18-74d1a288c240" containerID="d471a4a05f405da9b00f0e625376eea399442567c9a39bc49b5bd532152afe5f" exitCode=0
Jul 14 15:49:13.791271 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:13.790890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc" event={"ID":"46967238-4e63-4fc8-be18-74d1a288c240","Type":"ContainerDied","Data":"d471a4a05f405da9b00f0e625376eea399442567c9a39bc49b5bd532152afe5f"}
Jul 14 15:49:13.791319 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:13.791302    2563 scope.go:117] "RemoveContainer" containerID="d471a4a05f405da9b00f0e625376eea399442567c9a39bc49b5bd532152afe5f"
Jul 14 15:49:13.792417 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:13.792387    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:49:13.792489 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:13.792471    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:49:13.792560 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:13.792538    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:49:14.797415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:14.797381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc" event={"ID":"46967238-4e63-4fc8-be18-74d1a288c240","Type":"ContainerStarted","Data":"5af335774410aea92b26f363e7d06a3b27814022f082368f09734a9d70943197"}
Jul 14 15:49:15.125970 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:15.125384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:49:15.125970 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:15.125544    2563 patch_prober.go:28] interesting pod/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Jul 14 15:49:15.125970 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:15.125629    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc" podUID="46967238-4e63-4fc8-be18-74d1a288c240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Jul 14 15:49:16.124752 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:16.124716    2563 patch_prober.go:28] interesting pod/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Jul 14 15:49:16.125304 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:16.125268    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc" podUID="46967238-4e63-4fc8-be18-74d1a288c240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Jul 14 15:49:16.701299 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:16.701266    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:49:17.124484 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:17.124369    2563 patch_prober.go:28] interesting pod/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Jul 14 15:49:17.124484 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:17.124436    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc" podUID="46967238-4e63-4fc8-be18-74d1a288c240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Jul 14 15:49:18.124959 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:18.124920    2563 patch_prober.go:28] interesting pod/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Jul 14 15:49:18.125924 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:18.125890    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc" podUID="46967238-4e63-4fc8-be18-74d1a288c240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Jul 14 15:49:19.128022 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:19.127992    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:49:19.128937 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:19.128917    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:49:22.210408 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:22.210381    2563 scope.go:117] "RemoveContainer" containerID="2bdbe1de3d2b773bf4a5838f5cdcb692e9d60c7bafa1839d04869f2a7ee96d4d"
Jul 14 15:49:22.840049 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:22.839972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8" event={"ID":"fa63db74-4edb-4a3f-9023-63ecb6009dc2","Type":"ContainerStarted","Data":"aaf2bc138371173004fdf90a155577679e0c4d9e785904b9dcb96d4ef37fd33d"}
Jul 14 15:49:23.848057 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:23.848016    2563 generic.go:356] "Generic (PLEG): container finished" podID="fa63db74-4edb-4a3f-9023-63ecb6009dc2" containerID="aaf2bc138371173004fdf90a155577679e0c4d9e785904b9dcb96d4ef37fd33d" exitCode=0
Jul 14 15:49:23.848478 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:23.848079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8" event={"ID":"fa63db74-4edb-4a3f-9023-63ecb6009dc2","Type":"ContainerDied","Data":"aaf2bc138371173004fdf90a155577679e0c4d9e785904b9dcb96d4ef37fd33d"}
Jul 14 15:49:23.848478 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:23.848141    2563 scope.go:117] "RemoveContainer" containerID="2bdbe1de3d2b773bf4a5838f5cdcb692e9d60c7bafa1839d04869f2a7ee96d4d"
Jul 14 15:49:23.870272 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:23.870247    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8"]
Jul 14 15:49:24.998990 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:24.998961    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8"
Jul 14 15:49:25.142932 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:25.142904    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa63db74-4edb-4a3f-9023-63ecb6009dc2-maas-api-service-ca\") pod \"fa63db74-4edb-4a3f-9023-63ecb6009dc2\" (UID: \"fa63db74-4edb-4a3f-9023-63ecb6009dc2\") "
Jul 14 15:49:25.143250 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:25.143227    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa63db74-4edb-4a3f-9023-63ecb6009dc2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa63db74-4edb-4a3f-9023-63ecb6009dc2" (UID: "fa63db74-4edb-4a3f-9023-63ecb6009dc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:49:25.244518 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:25.244465    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa63db74-4edb-4a3f-9023-63ecb6009dc2-maas-api-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:49:25.861086 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:25.861058    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8"
Jul 14 15:49:25.861275 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:25.861052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8" event={"ID":"fa63db74-4edb-4a3f-9023-63ecb6009dc2","Type":"ContainerDied","Data":"7779444574bc5bfb560853368fbb871f3d956688e506bc4683d00ac90539315b"}
Jul 14 15:49:25.861275 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:25.861160    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7779444574bc5bfb560853368fbb871f3d956688e506bc4683d00ac90539315b"
Jul 14 15:49:25.897243 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:25.897212    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8"]
Jul 14 15:49:25.903241 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:25.903217    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-6htc8"]
Jul 14 15:49:26.216595 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:26.216563    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa63db74-4edb-4a3f-9023-63ecb6009dc2" path="/var/lib/kubelet/pods/fa63db74-4edb-4a3f-9023-63ecb6009dc2/volumes"
Jul 14 15:49:56.407269 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.407233    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp"]
Jul 14 15:49:56.407765 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.407660    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa63db74-4edb-4a3f-9023-63ecb6009dc2" containerName="revoke-keys"
Jul 14 15:49:56.407765 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.407671    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa63db74-4edb-4a3f-9023-63ecb6009dc2" containerName="revoke-keys"
Jul 14 15:49:56.407765 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.407700    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa63db74-4edb-4a3f-9023-63ecb6009dc2" containerName="revoke-keys"
Jul 14 15:49:56.407765 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.407705    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa63db74-4edb-4a3f-9023-63ecb6009dc2" containerName="revoke-keys"
Jul 14 15:49:56.407765 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.407764    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa63db74-4edb-4a3f-9023-63ecb6009dc2" containerName="revoke-keys"
Jul 14 15:49:56.407929 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.407773    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa63db74-4edb-4a3f-9023-63ecb6009dc2" containerName="revoke-keys"
Jul 14 15:49:56.407929 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.407780    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa63db74-4edb-4a3f-9023-63ecb6009dc2" containerName="revoke-keys"
Jul 14 15:49:56.410438 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.410422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp"
Jul 14 15:49:56.413100 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.413082    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\""
Jul 14 15:49:56.417685 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.417658    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp"]
Jul 14 15:49:56.526001 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.525963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf553a2-1dda-439e-b05f-aa6cc3ca6f34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp\" (UID: \"abf553a2-1dda-439e-b05f-aa6cc3ca6f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp"
Jul 14 15:49:56.626960 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.626918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf553a2-1dda-439e-b05f-aa6cc3ca6f34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp\" (UID: \"abf553a2-1dda-439e-b05f-aa6cc3ca6f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp"
Jul 14 15:49:56.627642 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.627617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf553a2-1dda-439e-b05f-aa6cc3ca6f34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp\" (UID: \"abf553a2-1dda-439e-b05f-aa6cc3ca6f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp"
Jul 14 15:49:56.721963 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.721854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp"
Jul 14 15:49:56.859142 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:56.859115    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp"]
Jul 14 15:49:56.861122 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:49:56.861095    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf553a2_1dda_439e_b05f_aa6cc3ca6f34.slice/crio-e2830e462a64bf97f7658fe77786aa715672f4c14f937df0306c6095b84123ef WatchSource:0}: Error finding container e2830e462a64bf97f7658fe77786aa715672f4c14f937df0306c6095b84123ef: Status 404 returned error can't find the container with id e2830e462a64bf97f7658fe77786aa715672f4c14f937df0306c6095b84123ef
Jul 14 15:49:57.026236 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:57.026159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp" event={"ID":"abf553a2-1dda-439e-b05f-aa6cc3ca6f34","Type":"ContainerStarted","Data":"e2830e462a64bf97f7658fe77786aa715672f4c14f937df0306c6095b84123ef"}
Jul 14 15:49:58.032538 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:58.032487    2563 generic.go:356] "Generic (PLEG): container finished" podID="abf553a2-1dda-439e-b05f-aa6cc3ca6f34" containerID="bb663d95ce9094cbae2fc032bf637ca92bb8cc1b073dfe517cd668e89ad117ba" exitCode=0
Jul 14 15:49:58.032954 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:58.032574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp" event={"ID":"abf553a2-1dda-439e-b05f-aa6cc3ca6f34","Type":"ContainerDied","Data":"bb663d95ce9094cbae2fc032bf637ca92bb8cc1b073dfe517cd668e89ad117ba"}
Jul 14 15:49:59.173757 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:59.173733    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp"
Jul 14 15:49:59.252079 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:59.252049    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf553a2-1dda-439e-b05f-aa6cc3ca6f34-maas-api-service-ca\") pod \"abf553a2-1dda-439e-b05f-aa6cc3ca6f34\" (UID: \"abf553a2-1dda-439e-b05f-aa6cc3ca6f34\") "
Jul 14 15:49:59.252408 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:59.252384    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf553a2-1dda-439e-b05f-aa6cc3ca6f34-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abf553a2-1dda-439e-b05f-aa6cc3ca6f34" (UID: "abf553a2-1dda-439e-b05f-aa6cc3ca6f34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:49:59.352918 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:49:59.352832    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf553a2-1dda-439e-b05f-aa6cc3ca6f34-maas-api-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:50:00.043895 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:00.043868    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp"
Jul 14 15:50:00.044064 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:00.043902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp" event={"ID":"abf553a2-1dda-439e-b05f-aa6cc3ca6f34","Type":"ContainerDied","Data":"e2830e462a64bf97f7658fe77786aa715672f4c14f937df0306c6095b84123ef"}
Jul 14 15:50:00.044064 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:00.043951    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2830e462a64bf97f7658fe77786aa715672f4c14f937df0306c6095b84123ef"
Jul 14 15:50:06.977148 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:06.977116    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"]
Jul 14 15:50:06.977627 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:06.977323    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2" podUID="c78286f9-a66f-43b6-9932-c4396d7db0ca" containerName="maas-api" containerID="cri-o://5dc4c73c92cd80db944f76244eb15d639069cbe82428f00fd66ebe7832ab6df6" gracePeriod=30
Jul 14 15:50:07.234865 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:07.234807    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:50:07.328698 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:07.328671    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c78286f9-a66f-43b6-9932-c4396d7db0ca-maas-api-tls\") pod \"c78286f9-a66f-43b6-9932-c4396d7db0ca\" (UID: \"c78286f9-a66f-43b6-9932-c4396d7db0ca\") "
Jul 14 15:50:07.328871 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:07.328723    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78286f9-a66f-43b6-9932-c4396d7db0ca-openshift-service-ca\") pod \"c78286f9-a66f-43b6-9932-c4396d7db0ca\" (UID: \"c78286f9-a66f-43b6-9932-c4396d7db0ca\") "
Jul 14 15:50:07.328871 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:07.328764    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2wxg\" (UniqueName: \"kubernetes.io/projected/c78286f9-a66f-43b6-9932-c4396d7db0ca-kube-api-access-q2wxg\") pod \"c78286f9-a66f-43b6-9932-c4396d7db0ca\" (UID: \"c78286f9-a66f-43b6-9932-c4396d7db0ca\") "
Jul 14 15:50:07.329166 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:07.329128    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78286f9-a66f-43b6-9932-c4396d7db0ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c78286f9-a66f-43b6-9932-c4396d7db0ca" (UID: "c78286f9-a66f-43b6-9932-c4396d7db0ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:50:07.331013 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:07.330985    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c78286f9-a66f-43b6-9932-c4396d7db0ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c78286f9-a66f-43b6-9932-c4396d7db0ca" (UID: "c78286f9-a66f-43b6-9932-c4396d7db0ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:50:07.331094 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:07.331022    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78286f9-a66f-43b6-9932-c4396d7db0ca-kube-api-access-q2wxg" (OuterVolumeSpecName: "kube-api-access-q2wxg") pod "c78286f9-a66f-43b6-9932-c4396d7db0ca" (UID: "c78286f9-a66f-43b6-9932-c4396d7db0ca"). InnerVolumeSpecName "kube-api-access-q2wxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:50:07.430558 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:07.430527    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c78286f9-a66f-43b6-9932-c4396d7db0ca-maas-api-tls\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:50:07.430558 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:07.430558    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78286f9-a66f-43b6-9932-c4396d7db0ca-openshift-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:50:07.430761 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:07.430568    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2wxg\" (UniqueName: \"kubernetes.io/projected/c78286f9-a66f-43b6-9932-c4396d7db0ca-kube-api-access-q2wxg\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:50:08.088577 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:08.088541    2563 generic.go:356] "Generic (PLEG): container finished" podID="c78286f9-a66f-43b6-9932-c4396d7db0ca" containerID="5dc4c73c92cd80db944f76244eb15d639069cbe82428f00fd66ebe7832ab6df6" exitCode=0
Jul 14 15:50:08.088979 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:08.088597    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"
Jul 14 15:50:08.088979 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:08.088637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2" event={"ID":"c78286f9-a66f-43b6-9932-c4396d7db0ca","Type":"ContainerDied","Data":"5dc4c73c92cd80db944f76244eb15d639069cbe82428f00fd66ebe7832ab6df6"}
Jul 14 15:50:08.088979 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:08.088711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2" event={"ID":"c78286f9-a66f-43b6-9932-c4396d7db0ca","Type":"ContainerDied","Data":"f74fd1c512b0aa2fcf806ed9e35d20910c9ec59ec3acadcfbb4f8e84da68a06a"}
Jul 14 15:50:08.088979 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:08.088733    2563 scope.go:117] "RemoveContainer" containerID="5dc4c73c92cd80db944f76244eb15d639069cbe82428f00fd66ebe7832ab6df6"
Jul 14 15:50:08.100450 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:08.100434    2563 scope.go:117] "RemoveContainer" containerID="5dc4c73c92cd80db944f76244eb15d639069cbe82428f00fd66ebe7832ab6df6"
Jul 14 15:50:08.100739 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:50:08.100722    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc4c73c92cd80db944f76244eb15d639069cbe82428f00fd66ebe7832ab6df6\": container with ID starting with 5dc4c73c92cd80db944f76244eb15d639069cbe82428f00fd66ebe7832ab6df6 not found: ID does not exist" containerID="5dc4c73c92cd80db944f76244eb15d639069cbe82428f00fd66ebe7832ab6df6"
Jul 14 15:50:08.100786 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:08.100748    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc4c73c92cd80db944f76244eb15d639069cbe82428f00fd66ebe7832ab6df6"} err="failed to get container status \"5dc4c73c92cd80db944f76244eb15d639069cbe82428f00fd66ebe7832ab6df6\": rpc error: code = NotFound desc = could not find container \"5dc4c73c92cd80db944f76244eb15d639069cbe82428f00fd66ebe7832ab6df6\": container with ID starting with 5dc4c73c92cd80db944f76244eb15d639069cbe82428f00fd66ebe7832ab6df6 not found: ID does not exist"
Jul 14 15:50:08.118222 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:08.118195    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"]
Jul 14 15:50:08.123161 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:08.123138    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd6a0c0f-6bccd5fdd5-f4fc2"]
Jul 14 15:50:08.216701 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:08.216662    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c78286f9-a66f-43b6-9932-c4396d7db0ca" path="/var/lib/kubelet/pods/c78286f9-a66f-43b6-9932-c4396d7db0ca/volumes"
Jul 14 15:50:22.619457 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:22.619420    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"]
Jul 14 15:50:22.619886 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:22.619685    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc" podUID="46967238-4e63-4fc8-be18-74d1a288c240" containerName="istio-proxy" containerID="cri-o://5af335774410aea92b26f363e7d06a3b27814022f082368f09734a9d70943197" gracePeriod=30
Jul 14 15:50:22.947093 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:22.947054    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"]
Jul 14 15:50:22.947586 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:22.947567    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa63db74-4edb-4a3f-9023-63ecb6009dc2" containerName="revoke-keys"
Jul 14 15:50:22.947698 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:22.947587    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa63db74-4edb-4a3f-9023-63ecb6009dc2" containerName="revoke-keys"
Jul 14 15:50:22.947698 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:22.947635    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf553a2-1dda-439e-b05f-aa6cc3ca6f34" containerName="revoke-keys"
Jul 14 15:50:22.947698 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:22.947643    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf553a2-1dda-439e-b05f-aa6cc3ca6f34" containerName="revoke-keys"
Jul 14 15:50:22.947698 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:22.947663    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c78286f9-a66f-43b6-9932-c4396d7db0ca" containerName="maas-api"
Jul 14 15:50:22.947698 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:22.947672    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c78286f9-a66f-43b6-9932-c4396d7db0ca" containerName="maas-api"
Jul 14 15:50:22.947937 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:22.947773    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c78286f9-a66f-43b6-9932-c4396d7db0ca" containerName="maas-api"
Jul 14 15:50:22.947937 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:22.947792    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf553a2-1dda-439e-b05f-aa6cc3ca6f34" containerName="revoke-keys"
Jul 14 15:50:22.950986 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:22.950966    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:22.953736 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:22.953717    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-390b960b-openshift-default-dockercfg-jhpdm\""
Jul 14 15:50:22.961549 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:22.961522    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"]
Jul 14 15:50:23.079105 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.079074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-istio-envoy\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.079280 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.079123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdxvb\" (UniqueName: \"kubernetes.io/projected/53feea43-52c0-449f-b475-b5f316f63400-kube-api-access-bdxvb\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.079280 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.079145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53feea43-52c0-449f-b475-b5f316f63400-istio-podinfo\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.079280 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.079203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-workload-certs\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.079392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.079285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53feea43-52c0-449f-b475-b5f316f63400-istiod-ca-cert\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.079392 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.079367    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-credential-socket\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.079453 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.079419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-workload-socket\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.079453 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.079438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-istio-data\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.079567 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.079465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53feea43-52c0-449f-b475-b5f316f63400-istio-token\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.179953 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.179925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-credential-socket\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.180149 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.179967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-workload-socket\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.180149 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.179984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-istio-data\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.180149 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.180001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53feea43-52c0-449f-b475-b5f316f63400-istio-token\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.180149 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.180032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-istio-envoy\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.180149 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.180072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdxvb\" (UniqueName: \"kubernetes.io/projected/53feea43-52c0-449f-b475-b5f316f63400-kube-api-access-bdxvb\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.180149 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.180093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53feea43-52c0-449f-b475-b5f316f63400-istio-podinfo\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.180149 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.180117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-workload-certs\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.180149 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.180148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53feea43-52c0-449f-b475-b5f316f63400-istiod-ca-cert\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.180771 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.180366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-workload-socket\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.180771 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.180405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-credential-socket\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.180771 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.180445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-istio-data\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.180771 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.180620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-workload-certs\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.180940 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.180838    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53feea43-52c0-449f-b475-b5f316f63400-istiod-ca-cert\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.182703 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.182672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-istio-envoy\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.182837 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.182820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53feea43-52c0-449f-b475-b5f316f63400-istio-podinfo\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.189718 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.189693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53feea43-52c0-449f-b475-b5f316f63400-istio-token\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.189846 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.189829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdxvb\" (UniqueName: \"kubernetes.io/projected/53feea43-52c0-449f-b475-b5f316f63400-kube-api-access-bdxvb\") pod \"e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") " pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.265020 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.264953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:23.406404 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.406380    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"]
Jul 14 15:50:23.408132 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:50:23.408106    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53feea43_52c0_449f_b475_b5f316f63400.slice/crio-e176f430579f68a89d036ae7b17cabdb1d70914a5c2cbd8e55e276313317c69d WatchSource:0}: Error finding container e176f430579f68a89d036ae7b17cabdb1d70914a5c2cbd8e55e276313317c69d: Status 404 returned error can't find the container with id e176f430579f68a89d036ae7b17cabdb1d70914a5c2cbd8e55e276313317c69d
Jul 14 15:50:23.410273 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.410241    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:50:23.410355 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.410305    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:50:23.410355 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:23.410335    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:50:24.186417 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.186386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c" event={"ID":"53feea43-52c0-449f-b475-b5f316f63400","Type":"ContainerStarted","Data":"7560a340a25c472af6111d957cbd2d2ca4e391bb0c510fc995b3b9f96299cac7"}
Jul 14 15:50:24.186805 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.186465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c" event={"ID":"53feea43-52c0-449f-b475-b5f316f63400","Type":"ContainerStarted","Data":"e176f430579f68a89d036ae7b17cabdb1d70914a5c2cbd8e55e276313317c69d"}
Jul 14 15:50:24.208700 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.208646    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c" podStartSLOduration=2.208632331 podStartE2EDuration="2.208632331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:50:24.207774778 +0000 UTC m=+1748.549040705" watchObservedRunningTime="2026-07-14 15:50:24.208632331 +0000 UTC m=+1748.549898238"
Jul 14 15:50:24.265780 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.265742    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:24.265989 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.265887    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"]
Jul 14 15:50:24.269781 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.269756    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:50:24.271378 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.271358    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:24.272545 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.272521    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npnj7\""
Jul 14 15:50:24.272545 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.272535    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-390b960b-serving-cert\""
Jul 14 15:50:24.272545 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.272547    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 15:50:24.272776 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.272499    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 15:50:24.278755 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.278728    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"]
Jul 14 15:50:24.392271 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.392234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc26l\" (UniqueName: \"kubernetes.io/projected/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-kube-api-access-pc26l\") pod \"maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp\" (UID: \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:50:24.392271 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.392282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-maas-api-tls\") pod \"maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp\" (UID: \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:50:24.392528 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.392362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-openshift-service-ca\") pod \"maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp\" (UID: \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:50:24.493300 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.493216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc26l\" (UniqueName: \"kubernetes.io/projected/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-kube-api-access-pc26l\") pod \"maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp\" (UID: \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:50:24.493300 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.493254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-maas-api-tls\") pod \"maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp\" (UID: \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:50:24.493558 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.493311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-openshift-service-ca\") pod \"maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp\" (UID: \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:50:24.494066 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.494039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-openshift-service-ca\") pod \"maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp\" (UID: \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:50:24.496157 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.496129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-maas-api-tls\") pod \"maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp\" (UID: \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:50:24.505050 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.505026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc26l\" (UniqueName: \"kubernetes.io/projected/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-kube-api-access-pc26l\") pod \"maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp\" (UID: \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:50:24.585613 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.585577    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:50:24.726554 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.726528    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"]
Jul 14 15:50:24.727961 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:50:24.727935    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f6ba18_a9c3_4fd7_a2db_df8d3629cf92.slice/crio-72435e8ba2b5ae2d16e05adcdc0cf6d57168d3a8a50858f58d2ebfb11a8b6fec WatchSource:0}: Error finding container 72435e8ba2b5ae2d16e05adcdc0cf6d57168d3a8a50858f58d2ebfb11a8b6fec: Status 404 returned error can't find the container with id 72435e8ba2b5ae2d16e05adcdc0cf6d57168d3a8a50858f58d2ebfb11a8b6fec
Jul 14 15:50:24.729449 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:24.729432    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 15:50:25.193971 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:25.193940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp" event={"ID":"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92","Type":"ContainerStarted","Data":"55847f66591eafe9ec3c0649e70146b2586624706e05ad13d1db7c3f43307176"}
Jul 14 15:50:25.194330 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:25.194000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp" event={"ID":"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92","Type":"ContainerStarted","Data":"72435e8ba2b5ae2d16e05adcdc0cf6d57168d3a8a50858f58d2ebfb11a8b6fec"}
Jul 14 15:50:25.194330 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:25.194028    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:50:25.194330 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:25.194214    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:25.195141 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:25.195124    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:25.212978 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:25.212934    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp" podStartSLOduration=0.825890864 podStartE2EDuration="1.212921321s" totalImagesPullingTime="387.030457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:50:25.211259055 +0000 UTC m=+1749.552524985" watchObservedRunningTime="2026-07-14 15:50:25.212921321 +0000 UTC m=+1749.554187228"
Jul 14 15:50:27.878696 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.878632    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:50:27.928458 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.928422    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-istio-envoy\") pod \"46967238-4e63-4fc8-be18-74d1a288c240\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") "
Jul 14 15:50:27.928670 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.928482    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46967238-4e63-4fc8-be18-74d1a288c240-istio-podinfo\") pod \"46967238-4e63-4fc8-be18-74d1a288c240\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") "
Jul 14 15:50:27.928670 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.928540    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpkck\" (UniqueName: \"kubernetes.io/projected/46967238-4e63-4fc8-be18-74d1a288c240-kube-api-access-zpkck\") pod \"46967238-4e63-4fc8-be18-74d1a288c240\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") "
Jul 14 15:50:27.928670 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.928572    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-istio-data\") pod \"46967238-4e63-4fc8-be18-74d1a288c240\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") "
Jul 14 15:50:27.928670 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.928600    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-workload-certs\") pod \"46967238-4e63-4fc8-be18-74d1a288c240\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") "
Jul 14 15:50:27.928972 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.928709    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46967238-4e63-4fc8-be18-74d1a288c240-istiod-ca-cert\") pod \"46967238-4e63-4fc8-be18-74d1a288c240\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") "
Jul 14 15:50:27.928972 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.928738    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46967238-4e63-4fc8-be18-74d1a288c240-istio-token\") pod \"46967238-4e63-4fc8-be18-74d1a288c240\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") "
Jul 14 15:50:27.928972 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.928786    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-workload-socket\") pod \"46967238-4e63-4fc8-be18-74d1a288c240\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") "
Jul 14 15:50:27.928972 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.928814    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-credential-socket\") pod \"46967238-4e63-4fc8-be18-74d1a288c240\" (UID: \"46967238-4e63-4fc8-be18-74d1a288c240\") "
Jul 14 15:50:27.929433 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.929234    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46967238-4e63-4fc8-be18-74d1a288c240-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "46967238-4e63-4fc8-be18-74d1a288c240" (UID: "46967238-4e63-4fc8-be18-74d1a288c240"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:50:27.929433 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.929309    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "46967238-4e63-4fc8-be18-74d1a288c240" (UID: "46967238-4e63-4fc8-be18-74d1a288c240"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:50:27.929433 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.929398    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "46967238-4e63-4fc8-be18-74d1a288c240" (UID: "46967238-4e63-4fc8-be18-74d1a288c240"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:50:27.929710 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.929496    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "46967238-4e63-4fc8-be18-74d1a288c240" (UID: "46967238-4e63-4fc8-be18-74d1a288c240"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:50:27.930352 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.930196    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-istio-data" (OuterVolumeSpecName: "istio-data") pod "46967238-4e63-4fc8-be18-74d1a288c240" (UID: "46967238-4e63-4fc8-be18-74d1a288c240"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:50:27.931383 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.931343    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "46967238-4e63-4fc8-be18-74d1a288c240" (UID: "46967238-4e63-4fc8-be18-74d1a288c240"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:50:27.931653 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.931629    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46967238-4e63-4fc8-be18-74d1a288c240-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "46967238-4e63-4fc8-be18-74d1a288c240" (UID: "46967238-4e63-4fc8-be18-74d1a288c240"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 15:50:27.932097 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.932070    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46967238-4e63-4fc8-be18-74d1a288c240-kube-api-access-zpkck" (OuterVolumeSpecName: "kube-api-access-zpkck") pod "46967238-4e63-4fc8-be18-74d1a288c240" (UID: "46967238-4e63-4fc8-be18-74d1a288c240"). InnerVolumeSpecName "kube-api-access-zpkck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:50:27.932385 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:27.932362    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46967238-4e63-4fc8-be18-74d1a288c240-istio-token" (OuterVolumeSpecName: "istio-token") pod "46967238-4e63-4fc8-be18-74d1a288c240" (UID: "46967238-4e63-4fc8-be18-74d1a288c240"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:50:28.029953 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.029924    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-workload-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:50:28.029953 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.029950    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-credential-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:50:28.029953 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.029961    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-istio-envoy\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:50:28.030165 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.029969    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46967238-4e63-4fc8-be18-74d1a288c240-istio-podinfo\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:50:28.030165 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.029978    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpkck\" (UniqueName: \"kubernetes.io/projected/46967238-4e63-4fc8-be18-74d1a288c240-kube-api-access-zpkck\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:50:28.030165 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.029986    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-istio-data\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:50:28.030165 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.029995    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46967238-4e63-4fc8-be18-74d1a288c240-workload-certs\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:50:28.030165 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.030002    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46967238-4e63-4fc8-be18-74d1a288c240-istiod-ca-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:50:28.030165 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.030010    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46967238-4e63-4fc8-be18-74d1a288c240-istio-token\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:50:28.211951 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.211916    2563 generic.go:356] "Generic (PLEG): container finished" podID="46967238-4e63-4fc8-be18-74d1a288c240" containerID="5af335774410aea92b26f363e7d06a3b27814022f082368f09734a9d70943197" exitCode=0
Jul 14 15:50:28.212126 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.212027    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"
Jul 14 15:50:28.218014 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.217988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc" event={"ID":"46967238-4e63-4fc8-be18-74d1a288c240","Type":"ContainerDied","Data":"5af335774410aea92b26f363e7d06a3b27814022f082368f09734a9d70943197"}
Jul 14 15:50:28.218132 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.218028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc" event={"ID":"46967238-4e63-4fc8-be18-74d1a288c240","Type":"ContainerDied","Data":"f21a6532c967681fd8196d35d1cb1f86f2db70b9f36e5b095073403886120c0b"}
Jul 14 15:50:28.218132 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.218058    2563 scope.go:117] "RemoveContainer" containerID="5af335774410aea92b26f363e7d06a3b27814022f082368f09734a9d70943197"
Jul 14 15:50:28.227631 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.227615    2563 scope.go:117] "RemoveContainer" containerID="d471a4a05f405da9b00f0e625376eea399442567c9a39bc49b5bd532152afe5f"
Jul 14 15:50:28.235771 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.235754    2563 scope.go:117] "RemoveContainer" containerID="5af335774410aea92b26f363e7d06a3b27814022f082368f09734a9d70943197"
Jul 14 15:50:28.236034 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:50:28.236013    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af335774410aea92b26f363e7d06a3b27814022f082368f09734a9d70943197\": container with ID starting with 5af335774410aea92b26f363e7d06a3b27814022f082368f09734a9d70943197 not found: ID does not exist" containerID="5af335774410aea92b26f363e7d06a3b27814022f082368f09734a9d70943197"
Jul 14 15:50:28.236083 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.236041    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af335774410aea92b26f363e7d06a3b27814022f082368f09734a9d70943197"} err="failed to get container status \"5af335774410aea92b26f363e7d06a3b27814022f082368f09734a9d70943197\": rpc error: code = NotFound desc = could not find container \"5af335774410aea92b26f363e7d06a3b27814022f082368f09734a9d70943197\": container with ID starting with 5af335774410aea92b26f363e7d06a3b27814022f082368f09734a9d70943197 not found: ID does not exist"
Jul 14 15:50:28.236083 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.236057    2563 scope.go:117] "RemoveContainer" containerID="d471a4a05f405da9b00f0e625376eea399442567c9a39bc49b5bd532152afe5f"
Jul 14 15:50:28.236289 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:50:28.236273    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d471a4a05f405da9b00f0e625376eea399442567c9a39bc49b5bd532152afe5f\": container with ID starting with d471a4a05f405da9b00f0e625376eea399442567c9a39bc49b5bd532152afe5f not found: ID does not exist" containerID="d471a4a05f405da9b00f0e625376eea399442567c9a39bc49b5bd532152afe5f"
Jul 14 15:50:28.236335 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.236295    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d471a4a05f405da9b00f0e625376eea399442567c9a39bc49b5bd532152afe5f"} err="failed to get container status \"d471a4a05f405da9b00f0e625376eea399442567c9a39bc49b5bd532152afe5f\": rpc error: code = NotFound desc = could not find container \"d471a4a05f405da9b00f0e625376eea399442567c9a39bc49b5bd532152afe5f\": container with ID starting with d471a4a05f405da9b00f0e625376eea399442567c9a39bc49b5bd532152afe5f not found: ID does not exist"
Jul 14 15:50:28.244635 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.244613    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"]
Jul 14 15:50:28.250730 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.250707    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-cd6a0c0f-openshift-default-585bd6fb85-fnfsc"]
Jul 14 15:50:28.548847 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.548758    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q"]
Jul 14 15:50:28.549195 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.549183    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46967238-4e63-4fc8-be18-74d1a288c240" containerName="istio-proxy"
Jul 14 15:50:28.549245 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.549197    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="46967238-4e63-4fc8-be18-74d1a288c240" containerName="istio-proxy"
Jul 14 15:50:28.549245 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.549207    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46967238-4e63-4fc8-be18-74d1a288c240" containerName="istio-proxy"
Jul 14 15:50:28.549245 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.549213    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="46967238-4e63-4fc8-be18-74d1a288c240" containerName="istio-proxy"
Jul 14 15:50:28.549352 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.549293    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="46967238-4e63-4fc8-be18-74d1a288c240" containerName="istio-proxy"
Jul 14 15:50:28.549352 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.549303    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="46967238-4e63-4fc8-be18-74d1a288c240" containerName="istio-proxy"
Jul 14 15:50:28.553871 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.553853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q"
Jul 14 15:50:28.557094 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.557071    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\""
Jul 14 15:50:28.559805 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.559751    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q"]
Jul 14 15:50:28.636264 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.636229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e4d2af-118c-4ebf-b661-86a2d6aaa391-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-390b960b-kfc9q\" (UID: \"17e4d2af-118c-4ebf-b661-86a2d6aaa391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q"
Jul 14 15:50:28.737253 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.737210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e4d2af-118c-4ebf-b661-86a2d6aaa391-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-390b960b-kfc9q\" (UID: \"17e4d2af-118c-4ebf-b661-86a2d6aaa391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q"
Jul 14 15:50:28.737882 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.737862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e4d2af-118c-4ebf-b661-86a2d6aaa391-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-390b960b-kfc9q\" (UID: \"17e4d2af-118c-4ebf-b661-86a2d6aaa391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q"
Jul 14 15:50:28.867623 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:28.867541    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q"
Jul 14 15:50:29.003414 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:29.003388    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q"]
Jul 14 15:50:29.005101 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:50:29.005074    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e4d2af_118c_4ebf_b661_86a2d6aaa391.slice/crio-2e4a11609974bd0d4997e9b6a540107cb07fd25c00eedcf7cfdd91c43711d88b WatchSource:0}: Error finding container 2e4a11609974bd0d4997e9b6a540107cb07fd25c00eedcf7cfdd91c43711d88b: Status 404 returned error can't find the container with id 2e4a11609974bd0d4997e9b6a540107cb07fd25c00eedcf7cfdd91c43711d88b
Jul 14 15:50:29.218564 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:29.218526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q" event={"ID":"17e4d2af-118c-4ebf-b661-86a2d6aaa391","Type":"ContainerStarted","Data":"2e4a11609974bd0d4997e9b6a540107cb07fd25c00eedcf7cfdd91c43711d88b"}
Jul 14 15:50:30.217001 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:30.216970    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46967238-4e63-4fc8-be18-74d1a288c240" path="/var/lib/kubelet/pods/46967238-4e63-4fc8-be18-74d1a288c240/volumes"
Jul 14 15:50:30.225437 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:30.225409    2563 generic.go:356] "Generic (PLEG): container finished" podID="17e4d2af-118c-4ebf-b661-86a2d6aaa391" containerID="d36e1979803e189fe5b0a38b74557db12ff8ffd0f25bff198e62ab8995bb135c" exitCode=7
Jul 14 15:50:30.225600 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:30.225446    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q" event={"ID":"17e4d2af-118c-4ebf-b661-86a2d6aaa391","Type":"ContainerDied","Data":"d36e1979803e189fe5b0a38b74557db12ff8ffd0f25bff198e62ab8995bb135c"}
Jul 14 15:50:30.225707 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:30.225688    2563 scope.go:117] "RemoveContainer" containerID="d36e1979803e189fe5b0a38b74557db12ff8ffd0f25bff198e62ab8995bb135c"
Jul 14 15:50:31.204768 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:31.204742    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:50:31.237123 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:31.237083    2563 generic.go:356] "Generic (PLEG): container finished" podID="17e4d2af-118c-4ebf-b661-86a2d6aaa391" containerID="2ec5853c0e38f77552327553a6593bdae63002bcd86976b7859e446e8d335bfe" exitCode=7
Jul 14 15:50:31.237605 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:31.237149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q" event={"ID":"17e4d2af-118c-4ebf-b661-86a2d6aaa391","Type":"ContainerDied","Data":"2ec5853c0e38f77552327553a6593bdae63002bcd86976b7859e446e8d335bfe"}
Jul 14 15:50:31.237605 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:31.237225    2563 scope.go:117] "RemoveContainer" containerID="d36e1979803e189fe5b0a38b74557db12ff8ffd0f25bff198e62ab8995bb135c"
Jul 14 15:50:31.237605 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:31.237437    2563 scope.go:117] "RemoveContainer" containerID="2ec5853c0e38f77552327553a6593bdae63002bcd86976b7859e446e8d335bfe"
Jul 14 15:50:31.237807 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:50:31.237780    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-390b960b-kfc9q_odh-ai-gateway-infra(17e4d2af-118c-4ebf-b661-86a2d6aaa391)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q" podUID="17e4d2af-118c-4ebf-b661-86a2d6aaa391"
Jul 14 15:50:32.244395 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:32.244372    2563 scope.go:117] "RemoveContainer" containerID="2ec5853c0e38f77552327553a6593bdae63002bcd86976b7859e446e8d335bfe"
Jul 14 15:50:32.244821 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:50:32.244627    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-390b960b-kfc9q_odh-ai-gateway-infra(17e4d2af-118c-4ebf-b661-86a2d6aaa391)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q" podUID="17e4d2af-118c-4ebf-b661-86a2d6aaa391"
Jul 14 15:50:40.195959 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:40.195863    2563 patch_prober.go:28] interesting pod/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 14 15:50:40.195959 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:40.195925    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c" podUID="53feea43-52c0-449f-b475-b5f316f63400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 14 15:50:40.292463 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:40.292429    2563 generic.go:356] "Generic (PLEG): container finished" podID="53feea43-52c0-449f-b475-b5f316f63400" containerID="7560a340a25c472af6111d957cbd2d2ca4e391bb0c510fc995b3b9f96299cac7" exitCode=0
Jul 14 15:50:40.292681 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:40.292524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c" event={"ID":"53feea43-52c0-449f-b475-b5f316f63400","Type":"ContainerDied","Data":"7560a340a25c472af6111d957cbd2d2ca4e391bb0c510fc995b3b9f96299cac7"}
Jul 14 15:50:40.292933 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:40.292914    2563 scope.go:117] "RemoveContainer" containerID="7560a340a25c472af6111d957cbd2d2ca4e391bb0c510fc995b3b9f96299cac7"
Jul 14 15:50:40.293943 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:40.293917    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:50:40.294011 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:40.293983    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:50:40.294049 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:40.294015    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:50:41.300430 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:41.300397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c" event={"ID":"53feea43-52c0-449f-b475-b5f316f63400","Type":"ContainerStarted","Data":"ef4bc3d716adbffe288fdabaaa0f1312bca97f792e71d047c92b419a91098dd4"}
Jul 14 15:50:42.265717 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:42.265678    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:42.265963 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:42.265940    2563 patch_prober.go:28] interesting pod/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 14 15:50:42.266046 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:42.265994    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c" podUID="53feea43-52c0-449f-b475-b5f316f63400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 14 15:50:43.265489 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:43.265449    2563 patch_prober.go:28] interesting pod/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 14 15:50:43.265907 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:43.265548    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c" podUID="53feea43-52c0-449f-b475-b5f316f63400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 14 15:50:44.265803 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:44.265764    2563 patch_prober.go:28] interesting pod/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 14 15:50:44.266197 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:44.265825    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c" podUID="53feea43-52c0-449f-b475-b5f316f63400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 14 15:50:45.269926 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:45.269896    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:45.270304 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:45.270170    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:45.270836 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:45.270818    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:50:47.210962 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:47.210921    2563 scope.go:117] "RemoveContainer" containerID="2ec5853c0e38f77552327553a6593bdae63002bcd86976b7859e446e8d335bfe"
Jul 14 15:50:48.344034 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:48.343948    2563 generic.go:356] "Generic (PLEG): container finished" podID="17e4d2af-118c-4ebf-b661-86a2d6aaa391" containerID="33bbc96d243cd0ac9449e040cdb39495439247c3a6435ec1d08e701228835c15" exitCode=0
Jul 14 15:50:48.344034 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:48.344021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q" event={"ID":"17e4d2af-118c-4ebf-b661-86a2d6aaa391","Type":"ContainerDied","Data":"33bbc96d243cd0ac9449e040cdb39495439247c3a6435ec1d08e701228835c15"}
Jul 14 15:50:48.344473 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:48.344072    2563 scope.go:117] "RemoveContainer" containerID="2ec5853c0e38f77552327553a6593bdae63002bcd86976b7859e446e8d335bfe"
Jul 14 15:50:49.377872 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:49.377842    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q"]
Jul 14 15:50:49.491916 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:49.491892    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q"
Jul 14 15:50:49.654746 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:49.654713    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e4d2af-118c-4ebf-b661-86a2d6aaa391-maas-api-service-ca\") pod \"17e4d2af-118c-4ebf-b661-86a2d6aaa391\" (UID: \"17e4d2af-118c-4ebf-b661-86a2d6aaa391\") "
Jul 14 15:50:49.655090 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:49.655066    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e4d2af-118c-4ebf-b661-86a2d6aaa391-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17e4d2af-118c-4ebf-b661-86a2d6aaa391" (UID: "17e4d2af-118c-4ebf-b661-86a2d6aaa391"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:50:49.755703 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:49.755666    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e4d2af-118c-4ebf-b661-86a2d6aaa391-maas-api-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:50:50.357957 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:50.357925    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q"
Jul 14 15:50:50.358142 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:50.357954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q" event={"ID":"17e4d2af-118c-4ebf-b661-86a2d6aaa391","Type":"ContainerDied","Data":"2e4a11609974bd0d4997e9b6a540107cb07fd25c00eedcf7cfdd91c43711d88b"}
Jul 14 15:50:50.358142 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:50.358021    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e4a11609974bd0d4997e9b6a540107cb07fd25c00eedcf7cfdd91c43711d88b"
Jul 14 15:50:50.390198 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:50.390169    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q"]
Jul 14 15:50:50.396721 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:50.396700    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-kfc9q"]
Jul 14 15:50:52.216681 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:50:52.216647    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17e4d2af-118c-4ebf-b661-86a2d6aaa391" path="/var/lib/kubelet/pods/17e4d2af-118c-4ebf-b661-86a2d6aaa391/volumes"
Jul 14 15:51:16.213176 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:16.213145    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:51:16.218419 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:16.218388    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:51:28.602134 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.602096    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49"]
Jul 14 15:51:28.602531 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.602518    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e4d2af-118c-4ebf-b661-86a2d6aaa391" containerName="revoke-keys"
Jul 14 15:51:28.602531 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.602529    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e4d2af-118c-4ebf-b661-86a2d6aaa391" containerName="revoke-keys"
Jul 14 15:51:28.602615 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.602546    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e4d2af-118c-4ebf-b661-86a2d6aaa391" containerName="revoke-keys"
Jul 14 15:51:28.602615 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.602552    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e4d2af-118c-4ebf-b661-86a2d6aaa391" containerName="revoke-keys"
Jul 14 15:51:28.602615 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.602558    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e4d2af-118c-4ebf-b661-86a2d6aaa391" containerName="revoke-keys"
Jul 14 15:51:28.602615 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.602564    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e4d2af-118c-4ebf-b661-86a2d6aaa391" containerName="revoke-keys"
Jul 14 15:51:28.602742 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.602634    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e4d2af-118c-4ebf-b661-86a2d6aaa391" containerName="revoke-keys"
Jul 14 15:51:28.602742 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.602642    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e4d2af-118c-4ebf-b661-86a2d6aaa391" containerName="revoke-keys"
Jul 14 15:51:28.605855 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.605841    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49"
Jul 14 15:51:28.608628 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.608608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\""
Jul 14 15:51:28.612993 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.612970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49"]
Jul 14 15:51:28.731230 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.731187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/523fa1a9-2d95-4a61-aa12-1c5060c86a20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-390b960b-hbl49\" (UID: \"523fa1a9-2d95-4a61-aa12-1c5060c86a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49"
Jul 14 15:51:28.832021 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.831992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/523fa1a9-2d95-4a61-aa12-1c5060c86a20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-390b960b-hbl49\" (UID: \"523fa1a9-2d95-4a61-aa12-1c5060c86a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49"
Jul 14 15:51:28.832646 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.832624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/523fa1a9-2d95-4a61-aa12-1c5060c86a20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-390b960b-hbl49\" (UID: \"523fa1a9-2d95-4a61-aa12-1c5060c86a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49"
Jul 14 15:51:28.918045 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:28.918016    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49"
Jul 14 15:51:29.052980 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:29.052955    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49"]
Jul 14 15:51:29.055309 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:51:29.055275    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523fa1a9_2d95_4a61_aa12_1c5060c86a20.slice/crio-74b0b542035cf368a2e706f477b8d7286953028bb008760a0180a9574b4c54c0 WatchSource:0}: Error finding container 74b0b542035cf368a2e706f477b8d7286953028bb008760a0180a9574b4c54c0: Status 404 returned error can't find the container with id 74b0b542035cf368a2e706f477b8d7286953028bb008760a0180a9574b4c54c0
Jul 14 15:51:29.572344 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:29.572254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49" event={"ID":"523fa1a9-2d95-4a61-aa12-1c5060c86a20","Type":"ContainerStarted","Data":"9d88ec4bceb070a57679781bb8e18fb4ed60b4db7d4d86073504056bfba42c14"}
Jul 14 15:51:29.572344 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:29.572322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49" event={"ID":"523fa1a9-2d95-4a61-aa12-1c5060c86a20","Type":"ContainerStarted","Data":"74b0b542035cf368a2e706f477b8d7286953028bb008760a0180a9574b4c54c0"}
Jul 14 15:51:29.590876 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:29.590830    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49" podStartSLOduration=1.590816579 podStartE2EDuration="1.590816579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:51:29.589713843 +0000 UTC m=+1813.930979752" watchObservedRunningTime="2026-07-14 15:51:29.590816579 +0000 UTC m=+1813.932082485"
Jul 14 15:51:30.579286 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:30.579252    2563 generic.go:356] "Generic (PLEG): container finished" podID="523fa1a9-2d95-4a61-aa12-1c5060c86a20" containerID="9d88ec4bceb070a57679781bb8e18fb4ed60b4db7d4d86073504056bfba42c14" exitCode=0
Jul 14 15:51:30.579711 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:30.579335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49" event={"ID":"523fa1a9-2d95-4a61-aa12-1c5060c86a20","Type":"ContainerDied","Data":"9d88ec4bceb070a57679781bb8e18fb4ed60b4db7d4d86073504056bfba42c14"}
Jul 14 15:51:31.721306 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:31.721283    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49"
Jul 14 15:51:31.861199 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:31.861103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/523fa1a9-2d95-4a61-aa12-1c5060c86a20-maas-api-service-ca\") pod \"523fa1a9-2d95-4a61-aa12-1c5060c86a20\" (UID: \"523fa1a9-2d95-4a61-aa12-1c5060c86a20\") "
Jul 14 15:51:31.861484 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:31.861461    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523fa1a9-2d95-4a61-aa12-1c5060c86a20-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "523fa1a9-2d95-4a61-aa12-1c5060c86a20" (UID: "523fa1a9-2d95-4a61-aa12-1c5060c86a20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:51:31.962648 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:31.962619    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/523fa1a9-2d95-4a61-aa12-1c5060c86a20-maas-api-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:51:32.593013 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:32.592931    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49"
Jul 14 15:51:32.593013 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:32.592940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-390b960b-hbl49" event={"ID":"523fa1a9-2d95-4a61-aa12-1c5060c86a20","Type":"ContainerDied","Data":"74b0b542035cf368a2e706f477b8d7286953028bb008760a0180a9574b4c54c0"}
Jul 14 15:51:32.593013 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:32.592984    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74b0b542035cf368a2e706f477b8d7286953028bb008760a0180a9574b4c54c0"
Jul 14 15:51:39.117961 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.117917    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"]
Jul 14 15:51:39.118431 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.118180    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp" podUID="28f6ba18-a9c3-4fd7-a2db-df8d3629cf92" containerName="maas-api" containerID="cri-o://55847f66591eafe9ec3c0649e70146b2586624706e05ad13d1db7c3f43307176" gracePeriod=30
Jul 14 15:51:39.369497 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.369437    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:51:39.438789 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.438758    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-maas-api-tls\") pod \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\" (UID: \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\") "
Jul 14 15:51:39.438944 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.438803    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc26l\" (UniqueName: \"kubernetes.io/projected/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-kube-api-access-pc26l\") pod \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\" (UID: \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\") "
Jul 14 15:51:39.438944 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.438856    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-openshift-service-ca\") pod \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\" (UID: \"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92\") "
Jul 14 15:51:39.439267 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.439235    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "28f6ba18-a9c3-4fd7-a2db-df8d3629cf92" (UID: "28f6ba18-a9c3-4fd7-a2db-df8d3629cf92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:51:39.441054 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.441032    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-kube-api-access-pc26l" (OuterVolumeSpecName: "kube-api-access-pc26l") pod "28f6ba18-a9c3-4fd7-a2db-df8d3629cf92" (UID: "28f6ba18-a9c3-4fd7-a2db-df8d3629cf92"). InnerVolumeSpecName "kube-api-access-pc26l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:51:39.441156 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.441105    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "28f6ba18-a9c3-4fd7-a2db-df8d3629cf92" (UID: "28f6ba18-a9c3-4fd7-a2db-df8d3629cf92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:51:39.539905 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.539874    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc26l\" (UniqueName: \"kubernetes.io/projected/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-kube-api-access-pc26l\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:51:39.539905 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.539900    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-openshift-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:51:39.539905 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.539912    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92-maas-api-tls\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:51:39.631170 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.631089    2563 generic.go:356] "Generic (PLEG): container finished" podID="28f6ba18-a9c3-4fd7-a2db-df8d3629cf92" containerID="55847f66591eafe9ec3c0649e70146b2586624706e05ad13d1db7c3f43307176" exitCode=0
Jul 14 15:51:39.631170 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.631130    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"
Jul 14 15:51:39.631349 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.631169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp" event={"ID":"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92","Type":"ContainerDied","Data":"55847f66591eafe9ec3c0649e70146b2586624706e05ad13d1db7c3f43307176"}
Jul 14 15:51:39.631349 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.631227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp" event={"ID":"28f6ba18-a9c3-4fd7-a2db-df8d3629cf92","Type":"ContainerDied","Data":"72435e8ba2b5ae2d16e05adcdc0cf6d57168d3a8a50858f58d2ebfb11a8b6fec"}
Jul 14 15:51:39.631349 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.631250    2563 scope.go:117] "RemoveContainer" containerID="55847f66591eafe9ec3c0649e70146b2586624706e05ad13d1db7c3f43307176"
Jul 14 15:51:39.641524 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.641485    2563 scope.go:117] "RemoveContainer" containerID="55847f66591eafe9ec3c0649e70146b2586624706e05ad13d1db7c3f43307176"
Jul 14 15:51:39.641777 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:51:39.641760    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55847f66591eafe9ec3c0649e70146b2586624706e05ad13d1db7c3f43307176\": container with ID starting with 55847f66591eafe9ec3c0649e70146b2586624706e05ad13d1db7c3f43307176 not found: ID does not exist" containerID="55847f66591eafe9ec3c0649e70146b2586624706e05ad13d1db7c3f43307176"
Jul 14 15:51:39.641835 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.641784    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55847f66591eafe9ec3c0649e70146b2586624706e05ad13d1db7c3f43307176"} err="failed to get container status \"55847f66591eafe9ec3c0649e70146b2586624706e05ad13d1db7c3f43307176\": rpc error: code = NotFound desc = could not find container \"55847f66591eafe9ec3c0649e70146b2586624706e05ad13d1db7c3f43307176\": container with ID starting with 55847f66591eafe9ec3c0649e70146b2586624706e05ad13d1db7c3f43307176 not found: ID does not exist"
Jul 14 15:51:39.659228 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.659191    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"]
Jul 14 15:51:39.673183 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:39.673157    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-390b960b-7c5b69c5db-hv6vp"]
Jul 14 15:51:40.216365 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:40.216327    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28f6ba18-a9c3-4fd7-a2db-df8d3629cf92" path="/var/lib/kubelet/pods/28f6ba18-a9c3-4fd7-a2db-df8d3629cf92/volumes"
Jul 14 15:51:53.919053 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:53.919018    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"]
Jul 14 15:51:53.919652 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:53.919231    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c" podUID="53feea43-52c0-449f-b475-b5f316f63400" containerName="istio-proxy" containerID="cri-o://ef4bc3d716adbffe288fdabaaa0f1312bca97f792e71d047c92b419a91098dd4" gracePeriod=30
Jul 14 15:51:59.182578 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.182555    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:51:59.332775 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.332744    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-istio-envoy\") pod \"53feea43-52c0-449f-b475-b5f316f63400\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") "
Jul 14 15:51:59.332775 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.332774    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-workload-socket\") pod \"53feea43-52c0-449f-b475-b5f316f63400\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") "
Jul 14 15:51:59.333010 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.332812    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53feea43-52c0-449f-b475-b5f316f63400-istiod-ca-cert\") pod \"53feea43-52c0-449f-b475-b5f316f63400\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") "
Jul 14 15:51:59.333010 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.332833    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53feea43-52c0-449f-b475-b5f316f63400-istio-token\") pod \"53feea43-52c0-449f-b475-b5f316f63400\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") "
Jul 14 15:51:59.333010 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.332852    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdxvb\" (UniqueName: \"kubernetes.io/projected/53feea43-52c0-449f-b475-b5f316f63400-kube-api-access-bdxvb\") pod \"53feea43-52c0-449f-b475-b5f316f63400\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") "
Jul 14 15:51:59.333010 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.332889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-credential-socket\") pod \"53feea43-52c0-449f-b475-b5f316f63400\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") "
Jul 14 15:51:59.333010 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.332906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-workload-certs\") pod \"53feea43-52c0-449f-b475-b5f316f63400\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") "
Jul 14 15:51:59.333010 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.332933    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-istio-data\") pod \"53feea43-52c0-449f-b475-b5f316f63400\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") "
Jul 14 15:51:59.333010 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.332964    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53feea43-52c0-449f-b475-b5f316f63400-istio-podinfo\") pod \"53feea43-52c0-449f-b475-b5f316f63400\" (UID: \"53feea43-52c0-449f-b475-b5f316f63400\") "
Jul 14 15:51:59.333369 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.333055    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "53feea43-52c0-449f-b475-b5f316f63400" (UID: "53feea43-52c0-449f-b475-b5f316f63400"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:51:59.333369 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.333168    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53feea43-52c0-449f-b475-b5f316f63400-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "53feea43-52c0-449f-b475-b5f316f63400" (UID: "53feea43-52c0-449f-b475-b5f316f63400"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:51:59.333369 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.333307    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-workload-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:51:59.333369 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.333325    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53feea43-52c0-449f-b475-b5f316f63400-istiod-ca-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:51:59.333831 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.333806    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "53feea43-52c0-449f-b475-b5f316f63400" (UID: "53feea43-52c0-449f-b475-b5f316f63400"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:51:59.334001 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.333920    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-istio-data" (OuterVolumeSpecName: "istio-data") pod "53feea43-52c0-449f-b475-b5f316f63400" (UID: "53feea43-52c0-449f-b475-b5f316f63400"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:51:59.334001 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.333990    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "53feea43-52c0-449f-b475-b5f316f63400" (UID: "53feea43-52c0-449f-b475-b5f316f63400"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:51:59.335083 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.335058    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53feea43-52c0-449f-b475-b5f316f63400-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "53feea43-52c0-449f-b475-b5f316f63400" (UID: "53feea43-52c0-449f-b475-b5f316f63400"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 15:51:59.335545 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.335525    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53feea43-52c0-449f-b475-b5f316f63400-istio-token" (OuterVolumeSpecName: "istio-token") pod "53feea43-52c0-449f-b475-b5f316f63400" (UID: "53feea43-52c0-449f-b475-b5f316f63400"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:51:59.335545 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.335532    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "53feea43-52c0-449f-b475-b5f316f63400" (UID: "53feea43-52c0-449f-b475-b5f316f63400"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:51:59.335870 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.335848    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53feea43-52c0-449f-b475-b5f316f63400-kube-api-access-bdxvb" (OuterVolumeSpecName: "kube-api-access-bdxvb") pod "53feea43-52c0-449f-b475-b5f316f63400" (UID: "53feea43-52c0-449f-b475-b5f316f63400"). InnerVolumeSpecName "kube-api-access-bdxvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:51:59.434439 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.434400    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53feea43-52c0-449f-b475-b5f316f63400-istio-podinfo\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:51:59.434439 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.434436    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-istio-envoy\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:51:59.434439 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.434445    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53feea43-52c0-449f-b475-b5f316f63400-istio-token\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:51:59.434653 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.434453    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdxvb\" (UniqueName: \"kubernetes.io/projected/53feea43-52c0-449f-b475-b5f316f63400-kube-api-access-bdxvb\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:51:59.434653 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.434462    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-credential-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:51:59.434653 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.434470    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-workload-certs\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:51:59.434653 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.434477    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53feea43-52c0-449f-b475-b5f316f63400-istio-data\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:51:59.744653 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.744616    2563 generic.go:356] "Generic (PLEG): container finished" podID="53feea43-52c0-449f-b475-b5f316f63400" containerID="ef4bc3d716adbffe288fdabaaa0f1312bca97f792e71d047c92b419a91098dd4" exitCode=0
Jul 14 15:51:59.744844 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.744678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c" event={"ID":"53feea43-52c0-449f-b475-b5f316f63400","Type":"ContainerDied","Data":"ef4bc3d716adbffe288fdabaaa0f1312bca97f792e71d047c92b419a91098dd4"}
Jul 14 15:51:59.744844 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.744681    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"
Jul 14 15:51:59.744844 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.744771    2563 scope.go:117] "RemoveContainer" containerID="ef4bc3d716adbffe288fdabaaa0f1312bca97f792e71d047c92b419a91098dd4"
Jul 14 15:51:59.744996 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.744757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c" event={"ID":"53feea43-52c0-449f-b475-b5f316f63400","Type":"ContainerDied","Data":"e176f430579f68a89d036ae7b17cabdb1d70914a5c2cbd8e55e276313317c69d"}
Jul 14 15:51:59.754227 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.754208    2563 scope.go:117] "RemoveContainer" containerID="7560a340a25c472af6111d957cbd2d2ca4e391bb0c510fc995b3b9f96299cac7"
Jul 14 15:51:59.762630 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.762599    2563 scope.go:117] "RemoveContainer" containerID="ef4bc3d716adbffe288fdabaaa0f1312bca97f792e71d047c92b419a91098dd4"
Jul 14 15:51:59.762873 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:51:59.762855    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4bc3d716adbffe288fdabaaa0f1312bca97f792e71d047c92b419a91098dd4\": container with ID starting with ef4bc3d716adbffe288fdabaaa0f1312bca97f792e71d047c92b419a91098dd4 not found: ID does not exist" containerID="ef4bc3d716adbffe288fdabaaa0f1312bca97f792e71d047c92b419a91098dd4"
Jul 14 15:51:59.762926 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.762881    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4bc3d716adbffe288fdabaaa0f1312bca97f792e71d047c92b419a91098dd4"} err="failed to get container status \"ef4bc3d716adbffe288fdabaaa0f1312bca97f792e71d047c92b419a91098dd4\": rpc error: code = NotFound desc = could not find container \"ef4bc3d716adbffe288fdabaaa0f1312bca97f792e71d047c92b419a91098dd4\": container with ID starting with ef4bc3d716adbffe288fdabaaa0f1312bca97f792e71d047c92b419a91098dd4 not found: ID does not exist"
Jul 14 15:51:59.762926 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.762896    2563 scope.go:117] "RemoveContainer" containerID="7560a340a25c472af6111d957cbd2d2ca4e391bb0c510fc995b3b9f96299cac7"
Jul 14 15:51:59.763130 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:51:59.763113    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7560a340a25c472af6111d957cbd2d2ca4e391bb0c510fc995b3b9f96299cac7\": container with ID starting with 7560a340a25c472af6111d957cbd2d2ca4e391bb0c510fc995b3b9f96299cac7 not found: ID does not exist" containerID="7560a340a25c472af6111d957cbd2d2ca4e391bb0c510fc995b3b9f96299cac7"
Jul 14 15:51:59.763186 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.763138    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7560a340a25c472af6111d957cbd2d2ca4e391bb0c510fc995b3b9f96299cac7"} err="failed to get container status \"7560a340a25c472af6111d957cbd2d2ca4e391bb0c510fc995b3b9f96299cac7\": rpc error: code = NotFound desc = could not find container \"7560a340a25c472af6111d957cbd2d2ca4e391bb0c510fc995b3b9f96299cac7\": container with ID starting with 7560a340a25c472af6111d957cbd2d2ca4e391bb0c510fc995b3b9f96299cac7 not found: ID does not exist"
Jul 14 15:51:59.767685 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.767664    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"]
Jul 14 15:51:59.772974 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:51:59.772956    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-390b960b-openshift-default-cd5fb94ff-r6x4c"]
Jul 14 15:52:00.216804 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:52:00.216773    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53feea43-52c0-449f-b475-b5f316f63400" path="/var/lib/kubelet/pods/53feea43-52c0-449f-b475-b5f316f63400/volumes"
Jul 14 15:53:16.320848 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:16.320814    2563 scope.go:117] "RemoveContainer" containerID="d2361754d35410d674ae35d719d9869b79759ee152a36476f0df3ff932b9191b"
Jul 14 15:53:34.027575 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.027537    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"]
Jul 14 15:53:34.028014 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.027940    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53feea43-52c0-449f-b475-b5f316f63400" containerName="istio-proxy"
Jul 14 15:53:34.028014 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.027952    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="53feea43-52c0-449f-b475-b5f316f63400" containerName="istio-proxy"
Jul 14 15:53:34.028014 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.027963    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53feea43-52c0-449f-b475-b5f316f63400" containerName="istio-proxy"
Jul 14 15:53:34.028014 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.027968    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="53feea43-52c0-449f-b475-b5f316f63400" containerName="istio-proxy"
Jul 14 15:53:34.028014 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.027976    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28f6ba18-a9c3-4fd7-a2db-df8d3629cf92" containerName="maas-api"
Jul 14 15:53:34.028014 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.027981    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="28f6ba18-a9c3-4fd7-a2db-df8d3629cf92" containerName="maas-api"
Jul 14 15:53:34.028014 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.028005    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="523fa1a9-2d95-4a61-aa12-1c5060c86a20" containerName="revoke-keys"
Jul 14 15:53:34.028014 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.028010    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="523fa1a9-2d95-4a61-aa12-1c5060c86a20" containerName="revoke-keys"
Jul 14 15:53:34.028271 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.028062    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="53feea43-52c0-449f-b475-b5f316f63400" containerName="istio-proxy"
Jul 14 15:53:34.028271 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.028068    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="523fa1a9-2d95-4a61-aa12-1c5060c86a20" containerName="revoke-keys"
Jul 14 15:53:34.028271 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.028076    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e4d2af-118c-4ebf-b661-86a2d6aaa391" containerName="revoke-keys"
Jul 14 15:53:34.028271 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.028083    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="28f6ba18-a9c3-4fd7-a2db-df8d3629cf92" containerName="maas-api"
Jul 14 15:53:34.031331 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.031314    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.034162 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.034140    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 15:53:34.034286 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.034217    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 15:53:34.034286 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.034250    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9cd0814a-openshift-default-dockercfg-ss92z\""
Jul 14 15:53:34.034397 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.034371    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 15:53:34.048198 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.048178    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"]
Jul 14 15:53:34.148496 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.148463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-podinfo\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.148496 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.148521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istiod-ca-cert\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.148725 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.148565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-workload-certs\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.148725 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.148603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-credential-socket\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.148725 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.148620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-token\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.148725 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.148635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-envoy\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.148725 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.148685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-data\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.148725 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.148705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmstc\" (UniqueName: \"kubernetes.io/projected/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-kube-api-access-hmstc\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.148941 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.148745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-workload-socket\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.249853 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.249816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-data\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.250061 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.249864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmstc\" (UniqueName: \"kubernetes.io/projected/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-kube-api-access-hmstc\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.250061 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.249906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-workload-socket\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.250061 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.249963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-podinfo\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.250061 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.250000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istiod-ca-cert\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.250296 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.250068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-workload-certs\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.250296 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.250127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-credential-socket\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.250296 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.250151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-token\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.250296 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.250174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-envoy\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.250757 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.250732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-credential-socket\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.250868 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.250755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-workload-certs\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.250928 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.250882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istiod-ca-cert\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.250986 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.250965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-workload-socket\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.251141 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.251107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-data\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.252567 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.252548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-envoy\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.252950 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.252934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-podinfo\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.262644 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.262618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmstc\" (UniqueName: \"kubernetes.io/projected/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-kube-api-access-hmstc\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.262776 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.262719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-token\") pod \"e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") " pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.341761 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.341672    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:34.483648 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.483622    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"]
Jul 14 15:53:34.485559 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:53:34.485497    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b96ac2a_0cd1_4753_8135_8de83a2a1de4.slice/crio-66080e2260ae86dfefa91db74930351a406be08cd281841a07c4521727661fbd WatchSource:0}: Error finding container 66080e2260ae86dfefa91db74930351a406be08cd281841a07c4521727661fbd: Status 404 returned error can't find the container with id 66080e2260ae86dfefa91db74930351a406be08cd281841a07c4521727661fbd
Jul 14 15:53:34.487678 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.487650    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:53:34.487734 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.487714    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:53:34.487783 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:34.487746    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:53:35.267173 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:35.267137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt" event={"ID":"3b96ac2a-0cd1-4753-8135-8de83a2a1de4","Type":"ContainerStarted","Data":"163408920e5dfddc34b3cd4ccb8c94dbc4823fea7d72e7515e1ea2c6480bff77"}
Jul 14 15:53:35.267654 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:35.267198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt" event={"ID":"3b96ac2a-0cd1-4753-8135-8de83a2a1de4","Type":"ContainerStarted","Data":"66080e2260ae86dfefa91db74930351a406be08cd281841a07c4521727661fbd"}
Jul 14 15:53:35.288788 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:35.288727    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt" podStartSLOduration=1.28870541 podStartE2EDuration="1.28870541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:53:35.288072503 +0000 UTC m=+1939.629338435" watchObservedRunningTime="2026-07-14 15:53:35.28870541 +0000 UTC m=+1939.629971319"
Jul 14 15:53:35.342424 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:35.342398    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:35.347362 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:35.347338    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:36.273001 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:36.272974    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:36.274018 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:36.273998    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:37.648430 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.648394    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"]
Jul 14 15:53:37.649189 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.649168    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="53feea43-52c0-449f-b475-b5f316f63400" containerName="istio-proxy"
Jul 14 15:53:37.653117 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.653093    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:53:37.656219 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.656195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9cd0814a-serving-cert\""
Jul 14 15:53:37.656356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.656333    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 15:53:37.656418 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.656358    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npnj7\""
Jul 14 15:53:37.656476 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.656341    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 15:53:37.661909 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.661887    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"]
Jul 14 15:53:37.784070 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.784042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5932082a-e704-4841-8b65-dd155f385e07-maas-api-tls\") pod \"maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc\" (UID: \"5932082a-e704-4841-8b65-dd155f385e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:53:37.784242 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.784082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5932082a-e704-4841-8b65-dd155f385e07-openshift-service-ca\") pod \"maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc\" (UID: \"5932082a-e704-4841-8b65-dd155f385e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:53:37.784242 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.784155    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlm5m\" (UniqueName: \"kubernetes.io/projected/5932082a-e704-4841-8b65-dd155f385e07-kube-api-access-rlm5m\") pod \"maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc\" (UID: \"5932082a-e704-4841-8b65-dd155f385e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:53:37.885224 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.885190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlm5m\" (UniqueName: \"kubernetes.io/projected/5932082a-e704-4841-8b65-dd155f385e07-kube-api-access-rlm5m\") pod \"maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc\" (UID: \"5932082a-e704-4841-8b65-dd155f385e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:53:37.885380 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.885292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5932082a-e704-4841-8b65-dd155f385e07-maas-api-tls\") pod \"maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc\" (UID: \"5932082a-e704-4841-8b65-dd155f385e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:53:37.885380 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.885330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5932082a-e704-4841-8b65-dd155f385e07-openshift-service-ca\") pod \"maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc\" (UID: \"5932082a-e704-4841-8b65-dd155f385e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:53:37.885456 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:53:37.885436    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-serving-cert: secret "maas-api-e2e-mt-9cd0814a-serving-cert" not found
Jul 14 15:53:37.885533 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:53:37.885521    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5932082a-e704-4841-8b65-dd155f385e07-maas-api-tls podName:5932082a-e704-4841-8b65-dd155f385e07 nodeName:}" failed. No retries permitted until 2026-07-14 15:53:38.38548465 +0000 UTC m=+1942.726750549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5932082a-e704-4841-8b65-dd155f385e07-maas-api-tls") pod "maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc" (UID: "5932082a-e704-4841-8b65-dd155f385e07") : secret "maas-api-e2e-mt-9cd0814a-serving-cert" not found
Jul 14 15:53:37.885931 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.885915    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5932082a-e704-4841-8b65-dd155f385e07-openshift-service-ca\") pod \"maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc\" (UID: \"5932082a-e704-4841-8b65-dd155f385e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:53:37.897512 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:37.897472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlm5m\" (UniqueName: \"kubernetes.io/projected/5932082a-e704-4841-8b65-dd155f385e07-kube-api-access-rlm5m\") pod \"maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc\" (UID: \"5932082a-e704-4841-8b65-dd155f385e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:53:38.391114 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:38.391083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5932082a-e704-4841-8b65-dd155f385e07-maas-api-tls\") pod \"maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc\" (UID: \"5932082a-e704-4841-8b65-dd155f385e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:53:38.393703 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:38.393680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5932082a-e704-4841-8b65-dd155f385e07-maas-api-tls\") pod \"maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc\" (UID: \"5932082a-e704-4841-8b65-dd155f385e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:53:38.566944 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:38.566914    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:53:38.706454 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:38.706427    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"]
Jul 14 15:53:38.708218 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:53:38.708187    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5932082a_e704_4841_8b65_dd155f385e07.slice/crio-3045cb1951b54c2506bb899a152bf69fdac5ac5e0779478e4d4e64283bfeadfe WatchSource:0}: Error finding container 3045cb1951b54c2506bb899a152bf69fdac5ac5e0779478e4d4e64283bfeadfe: Status 404 returned error can't find the container with id 3045cb1951b54c2506bb899a152bf69fdac5ac5e0779478e4d4e64283bfeadfe
Jul 14 15:53:39.289715 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:39.289679    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc" event={"ID":"5932082a-e704-4841-8b65-dd155f385e07","Type":"ContainerStarted","Data":"3045cb1951b54c2506bb899a152bf69fdac5ac5e0779478e4d4e64283bfeadfe"}
Jul 14 15:53:40.296980 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:40.296937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc" event={"ID":"5932082a-e704-4841-8b65-dd155f385e07","Type":"ContainerStarted","Data":"74bc8ce8941836fa378dac1267218c85557b175c3c471ee1a710341410f78598"}
Jul 14 15:53:40.297412 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:40.297017    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:53:40.315796 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:40.315749    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc" podStartSLOduration=2.264662161 podStartE2EDuration="3.315734311s" totalImagesPullingTime="1.05107215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:53:40.314273234 +0000 UTC m=+1944.655539144" watchObservedRunningTime="2026-07-14 15:53:40.315734311 +0000 UTC m=+1944.657000218"
Jul 14 15:53:46.308056 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:46.308024    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:53:50.093546 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:50.093493    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r"]
Jul 14 15:53:50.099384 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:50.099364    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r"
Jul 14 15:53:50.102751 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:50.102532    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\""
Jul 14 15:53:50.104892 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:50.104862    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r"]
Jul 14 15:53:50.208667 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:50.208628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b9e4de-5f94-4ff6-bfa5-9abe071cc174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r\" (UID: \"66b9e4de-5f94-4ff6-bfa5-9abe071cc174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r"
Jul 14 15:53:50.310150 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:50.310116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b9e4de-5f94-4ff6-bfa5-9abe071cc174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r\" (UID: \"66b9e4de-5f94-4ff6-bfa5-9abe071cc174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r"
Jul 14 15:53:50.310969 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:50.310940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b9e4de-5f94-4ff6-bfa5-9abe071cc174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r\" (UID: \"66b9e4de-5f94-4ff6-bfa5-9abe071cc174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r"
Jul 14 15:53:50.414542 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:50.414512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r"
Jul 14 15:53:50.557115 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:50.557082    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r"]
Jul 14 15:53:50.558928 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:53:50.558896    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b9e4de_5f94_4ff6_bfa5_9abe071cc174.slice/crio-709cdd7f810236056b6b43443897adfc840aad373fbb7eb067e7e11cc02fd909 WatchSource:0}: Error finding container 709cdd7f810236056b6b43443897adfc840aad373fbb7eb067e7e11cc02fd909: Status 404 returned error can't find the container with id 709cdd7f810236056b6b43443897adfc840aad373fbb7eb067e7e11cc02fd909
Jul 14 15:53:51.274391 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:51.274303    2563 patch_prober.go:28] interesting pod/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Jul 14 15:53:51.274391 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:51.274359    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt" podUID="3b96ac2a-0cd1-4753-8135-8de83a2a1de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Jul 14 15:53:51.376285 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:51.376248    2563 generic.go:356] "Generic (PLEG): container finished" podID="66b9e4de-5f94-4ff6-bfa5-9abe071cc174" containerID="98aed2793d8ad7ed13ee48e4d72a4c7f2c9af6f56dcac994441f74eaf3d10656" exitCode=0
Jul 14 15:53:51.376424 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:51.376332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r" event={"ID":"66b9e4de-5f94-4ff6-bfa5-9abe071cc174","Type":"ContainerDied","Data":"98aed2793d8ad7ed13ee48e4d72a4c7f2c9af6f56dcac994441f74eaf3d10656"}
Jul 14 15:53:51.376424 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:51.376384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r" event={"ID":"66b9e4de-5f94-4ff6-bfa5-9abe071cc174","Type":"ContainerStarted","Data":"709cdd7f810236056b6b43443897adfc840aad373fbb7eb067e7e11cc02fd909"}
Jul 14 15:53:51.377781 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:51.377758    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b96ac2a-0cd1-4753-8135-8de83a2a1de4" containerID="163408920e5dfddc34b3cd4ccb8c94dbc4823fea7d72e7515e1ea2c6480bff77" exitCode=0
Jul 14 15:53:51.377874 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:51.377797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt" event={"ID":"3b96ac2a-0cd1-4753-8135-8de83a2a1de4","Type":"ContainerDied","Data":"163408920e5dfddc34b3cd4ccb8c94dbc4823fea7d72e7515e1ea2c6480bff77"}
Jul 14 15:53:51.378166 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:51.378154    2563 scope.go:117] "RemoveContainer" containerID="163408920e5dfddc34b3cd4ccb8c94dbc4823fea7d72e7515e1ea2c6480bff77"
Jul 14 15:53:51.379164 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:51.379136    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:53:51.379250 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:51.379234    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:53:51.379287 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:51.379274    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:53:52.384625 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:52.384582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt" event={"ID":"3b96ac2a-0cd1-4753-8135-8de83a2a1de4","Type":"ContainerStarted","Data":"ca0b71349fe4adcc6ad3370e215d4d8725b641e43e13ac5edb41c16252b5e40f"}
Jul 14 15:53:52.557220 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:52.557191    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r"
Jul 14 15:53:52.642592 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:52.641938    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b9e4de-5f94-4ff6-bfa5-9abe071cc174-maas-api-service-ca\") pod \"66b9e4de-5f94-4ff6-bfa5-9abe071cc174\" (UID: \"66b9e4de-5f94-4ff6-bfa5-9abe071cc174\") "
Jul 14 15:53:52.642778 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:52.642679    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b9e4de-5f94-4ff6-bfa5-9abe071cc174-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "66b9e4de-5f94-4ff6-bfa5-9abe071cc174" (UID: "66b9e4de-5f94-4ff6-bfa5-9abe071cc174"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:53:52.743857 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:52.743818    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b9e4de-5f94-4ff6-bfa5-9abe071cc174-maas-api-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:53:53.342217 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:53.342176    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:53.342472 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:53.342440    2563 patch_prober.go:28] interesting pod/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Jul 14 15:53:53.342612 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:53.342521    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt" podUID="3b96ac2a-0cd1-4753-8135-8de83a2a1de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Jul 14 15:53:53.390842 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:53.390817    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r"
Jul 14 15:53:53.391210 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:53.390852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r" event={"ID":"66b9e4de-5f94-4ff6-bfa5-9abe071cc174","Type":"ContainerDied","Data":"709cdd7f810236056b6b43443897adfc840aad373fbb7eb067e7e11cc02fd909"}
Jul 14 15:53:53.391210 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:53.390898    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="709cdd7f810236056b6b43443897adfc840aad373fbb7eb067e7e11cc02fd909"
Jul 14 15:53:54.342706 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:54.342670    2563 patch_prober.go:28] interesting pod/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Jul 14 15:53:54.342997 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:54.342968    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt" podUID="3b96ac2a-0cd1-4753-8135-8de83a2a1de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Jul 14 15:53:55.342459 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:55.342419    2563 patch_prober.go:28] interesting pod/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Jul 14 15:53:55.343121 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:55.343086    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt" podUID="3b96ac2a-0cd1-4753-8135-8de83a2a1de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Jul 14 15:53:56.346264 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:56.346236    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:56.346674 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:56.346494    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:53:56.347319 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:53:56.347301    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:54:00.596442 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:00.596405    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"]
Jul 14 15:54:00.596961 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:00.596678    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc" podUID="5932082a-e704-4841-8b65-dd155f385e07" containerName="maas-api" containerID="cri-o://74bc8ce8941836fa378dac1267218c85557b175c3c471ee1a710341410f78598" gracePeriod=30
Jul 14 15:54:00.855402 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:00.855334    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:54:00.930143 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:00.930110    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5932082a-e704-4841-8b65-dd155f385e07-maas-api-tls\") pod \"5932082a-e704-4841-8b65-dd155f385e07\" (UID: \"5932082a-e704-4841-8b65-dd155f385e07\") "
Jul 14 15:54:00.930308 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:00.930164    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5932082a-e704-4841-8b65-dd155f385e07-openshift-service-ca\") pod \"5932082a-e704-4841-8b65-dd155f385e07\" (UID: \"5932082a-e704-4841-8b65-dd155f385e07\") "
Jul 14 15:54:00.930308 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:00.930192    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlm5m\" (UniqueName: \"kubernetes.io/projected/5932082a-e704-4841-8b65-dd155f385e07-kube-api-access-rlm5m\") pod \"5932082a-e704-4841-8b65-dd155f385e07\" (UID: \"5932082a-e704-4841-8b65-dd155f385e07\") "
Jul 14 15:54:00.930618 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:00.930592    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5932082a-e704-4841-8b65-dd155f385e07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5932082a-e704-4841-8b65-dd155f385e07" (UID: "5932082a-e704-4841-8b65-dd155f385e07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:54:00.932349 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:00.932328    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5932082a-e704-4841-8b65-dd155f385e07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5932082a-e704-4841-8b65-dd155f385e07" (UID: "5932082a-e704-4841-8b65-dd155f385e07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:54:00.932448 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:00.932427    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5932082a-e704-4841-8b65-dd155f385e07-kube-api-access-rlm5m" (OuterVolumeSpecName: "kube-api-access-rlm5m") pod "5932082a-e704-4841-8b65-dd155f385e07" (UID: "5932082a-e704-4841-8b65-dd155f385e07"). InnerVolumeSpecName "kube-api-access-rlm5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:54:01.031563 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:01.031527    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5932082a-e704-4841-8b65-dd155f385e07-maas-api-tls\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:54:01.031563 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:01.031558    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5932082a-e704-4841-8b65-dd155f385e07-openshift-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:54:01.031563 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:01.031567    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlm5m\" (UniqueName: \"kubernetes.io/projected/5932082a-e704-4841-8b65-dd155f385e07-kube-api-access-rlm5m\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:54:01.443055 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:01.443013    2563 generic.go:356] "Generic (PLEG): container finished" podID="5932082a-e704-4841-8b65-dd155f385e07" containerID="74bc8ce8941836fa378dac1267218c85557b175c3c471ee1a710341410f78598" exitCode=0
Jul 14 15:54:01.443220 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:01.443078    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"
Jul 14 15:54:01.443220 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:01.443107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc" event={"ID":"5932082a-e704-4841-8b65-dd155f385e07","Type":"ContainerDied","Data":"74bc8ce8941836fa378dac1267218c85557b175c3c471ee1a710341410f78598"}
Jul 14 15:54:01.443220 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:01.443174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc" event={"ID":"5932082a-e704-4841-8b65-dd155f385e07","Type":"ContainerDied","Data":"3045cb1951b54c2506bb899a152bf69fdac5ac5e0779478e4d4e64283bfeadfe"}
Jul 14 15:54:01.443344 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:01.443223    2563 scope.go:117] "RemoveContainer" containerID="74bc8ce8941836fa378dac1267218c85557b175c3c471ee1a710341410f78598"
Jul 14 15:54:01.453109 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:01.453091    2563 scope.go:117] "RemoveContainer" containerID="74bc8ce8941836fa378dac1267218c85557b175c3c471ee1a710341410f78598"
Jul 14 15:54:01.453397 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:54:01.453369    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bc8ce8941836fa378dac1267218c85557b175c3c471ee1a710341410f78598\": container with ID starting with 74bc8ce8941836fa378dac1267218c85557b175c3c471ee1a710341410f78598 not found: ID does not exist" containerID="74bc8ce8941836fa378dac1267218c85557b175c3c471ee1a710341410f78598"
Jul 14 15:54:01.453530 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:01.453397    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bc8ce8941836fa378dac1267218c85557b175c3c471ee1a710341410f78598"} err="failed to get container status \"74bc8ce8941836fa378dac1267218c85557b175c3c471ee1a710341410f78598\": rpc error: code = NotFound desc = could not find container \"74bc8ce8941836fa378dac1267218c85557b175c3c471ee1a710341410f78598\": container with ID starting with 74bc8ce8941836fa378dac1267218c85557b175c3c471ee1a710341410f78598 not found: ID does not exist"
Jul 14 15:54:01.467217 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:01.467187    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"]
Jul 14 15:54:01.477621 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:01.477599    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cd0814a-d88f9c8bc-f8psc"]
Jul 14 15:54:02.218959 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:02.218909    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5932082a-e704-4841-8b65-dd155f385e07" path="/var/lib/kubelet/pods/5932082a-e704-4841-8b65-dd155f385e07/volumes"
Jul 14 15:54:15.952557 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:15.952461    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"]
Jul 14 15:54:15.953044 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:15.952703    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt" podUID="3b96ac2a-0cd1-4753-8135-8de83a2a1de4" containerName="istio-proxy" containerID="cri-o://ca0b71349fe4adcc6ad3370e215d4d8725b641e43e13ac5edb41c16252b5e40f" gracePeriod=30
Jul 14 15:54:16.342232 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:16.342149    2563 scope.go:117] "RemoveContainer" containerID="dc1bc9b7fdd89347dc9de0f2f67fa0560051a10c3006182fdbb35616150560f9"
Jul 14 15:54:21.218529 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.218457    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:54:21.319155 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.319127    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-envoy\") pod \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") "
Jul 14 15:54:21.319349 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.319174    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-podinfo\") pod \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") "
Jul 14 15:54:21.319349 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.319218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-workload-certs\") pod \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") "
Jul 14 15:54:21.319349 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.319311    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-data\") pod \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") "
Jul 14 15:54:21.319558 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.319360    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmstc\" (UniqueName: \"kubernetes.io/projected/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-kube-api-access-hmstc\") pod \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") "
Jul 14 15:54:21.319558 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.319398    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-credential-socket\") pod \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") "
Jul 14 15:54:21.319558 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.319437    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-token\") pod \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") "
Jul 14 15:54:21.319558 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.319467    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istiod-ca-cert\") pod \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") "
Jul 14 15:54:21.319558 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.319541    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-workload-socket\") pod \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\" (UID: \"3b96ac2a-0cd1-4753-8135-8de83a2a1de4\") "
Jul 14 15:54:21.319842 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.319597    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b96ac2a-0cd1-4753-8135-8de83a2a1de4" (UID: "3b96ac2a-0cd1-4753-8135-8de83a2a1de4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:54:21.319842 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.319730    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b96ac2a-0cd1-4753-8135-8de83a2a1de4" (UID: "3b96ac2a-0cd1-4753-8135-8de83a2a1de4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:54:21.320117 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.319989    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-workload-certs\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:54:21.320117 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.320011    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-credential-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:54:21.320242 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.320189    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b96ac2a-0cd1-4753-8135-8de83a2a1de4" (UID: "3b96ac2a-0cd1-4753-8135-8de83a2a1de4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:54:21.320787 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.320765    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b96ac2a-0cd1-4753-8135-8de83a2a1de4" (UID: "3b96ac2a-0cd1-4753-8135-8de83a2a1de4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:54:21.321148 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.321115    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b96ac2a-0cd1-4753-8135-8de83a2a1de4" (UID: "3b96ac2a-0cd1-4753-8135-8de83a2a1de4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:54:21.321742 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.321716    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b96ac2a-0cd1-4753-8135-8de83a2a1de4" (UID: "3b96ac2a-0cd1-4753-8135-8de83a2a1de4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:54:21.322027 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.322003    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b96ac2a-0cd1-4753-8135-8de83a2a1de4" (UID: "3b96ac2a-0cd1-4753-8135-8de83a2a1de4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 15:54:21.322354 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.322315    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-kube-api-access-hmstc" (OuterVolumeSpecName: "kube-api-access-hmstc") pod "3b96ac2a-0cd1-4753-8135-8de83a2a1de4" (UID: "3b96ac2a-0cd1-4753-8135-8de83a2a1de4"). InnerVolumeSpecName "kube-api-access-hmstc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:54:21.322433 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.322372    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b96ac2a-0cd1-4753-8135-8de83a2a1de4" (UID: "3b96ac2a-0cd1-4753-8135-8de83a2a1de4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:54:21.421481 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.421447    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-workload-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:54:21.421481 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.421476    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-envoy\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:54:21.421712 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.421529    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-podinfo\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:54:21.421712 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.421541    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-data\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:54:21.421712 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.421554    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmstc\" (UniqueName: \"kubernetes.io/projected/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-kube-api-access-hmstc\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:54:21.421712 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.421565    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istio-token\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:54:21.421712 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.421578    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b96ac2a-0cd1-4753-8135-8de83a2a1de4-istiod-ca-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:54:21.552773 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.552685    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b96ac2a-0cd1-4753-8135-8de83a2a1de4" containerID="ca0b71349fe4adcc6ad3370e215d4d8725b641e43e13ac5edb41c16252b5e40f" exitCode=0
Jul 14 15:54:21.552773 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.552739    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"
Jul 14 15:54:21.552972 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.552773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt" event={"ID":"3b96ac2a-0cd1-4753-8135-8de83a2a1de4","Type":"ContainerDied","Data":"ca0b71349fe4adcc6ad3370e215d4d8725b641e43e13ac5edb41c16252b5e40f"}
Jul 14 15:54:21.552972 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.552832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt" event={"ID":"3b96ac2a-0cd1-4753-8135-8de83a2a1de4","Type":"ContainerDied","Data":"66080e2260ae86dfefa91db74930351a406be08cd281841a07c4521727661fbd"}
Jul 14 15:54:21.552972 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.552860    2563 scope.go:117] "RemoveContainer" containerID="ca0b71349fe4adcc6ad3370e215d4d8725b641e43e13ac5edb41c16252b5e40f"
Jul 14 15:54:21.563652 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.563477    2563 scope.go:117] "RemoveContainer" containerID="163408920e5dfddc34b3cd4ccb8c94dbc4823fea7d72e7515e1ea2c6480bff77"
Jul 14 15:54:21.573092 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.573002    2563 scope.go:117] "RemoveContainer" containerID="ca0b71349fe4adcc6ad3370e215d4d8725b641e43e13ac5edb41c16252b5e40f"
Jul 14 15:54:21.573357 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:54:21.573332    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0b71349fe4adcc6ad3370e215d4d8725b641e43e13ac5edb41c16252b5e40f\": container with ID starting with ca0b71349fe4adcc6ad3370e215d4d8725b641e43e13ac5edb41c16252b5e40f not found: ID does not exist" containerID="ca0b71349fe4adcc6ad3370e215d4d8725b641e43e13ac5edb41c16252b5e40f"
Jul 14 15:54:21.573446 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.573369    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0b71349fe4adcc6ad3370e215d4d8725b641e43e13ac5edb41c16252b5e40f"} err="failed to get container status \"ca0b71349fe4adcc6ad3370e215d4d8725b641e43e13ac5edb41c16252b5e40f\": rpc error: code = NotFound desc = could not find container \"ca0b71349fe4adcc6ad3370e215d4d8725b641e43e13ac5edb41c16252b5e40f\": container with ID starting with ca0b71349fe4adcc6ad3370e215d4d8725b641e43e13ac5edb41c16252b5e40f not found: ID does not exist"
Jul 14 15:54:21.573446 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.573392    2563 scope.go:117] "RemoveContainer" containerID="163408920e5dfddc34b3cd4ccb8c94dbc4823fea7d72e7515e1ea2c6480bff77"
Jul 14 15:54:21.573705 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:54:21.573685    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163408920e5dfddc34b3cd4ccb8c94dbc4823fea7d72e7515e1ea2c6480bff77\": container with ID starting with 163408920e5dfddc34b3cd4ccb8c94dbc4823fea7d72e7515e1ea2c6480bff77 not found: ID does not exist" containerID="163408920e5dfddc34b3cd4ccb8c94dbc4823fea7d72e7515e1ea2c6480bff77"
Jul 14 15:54:21.573822 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.573711    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163408920e5dfddc34b3cd4ccb8c94dbc4823fea7d72e7515e1ea2c6480bff77"} err="failed to get container status \"163408920e5dfddc34b3cd4ccb8c94dbc4823fea7d72e7515e1ea2c6480bff77\": rpc error: code = NotFound desc = could not find container \"163408920e5dfddc34b3cd4ccb8c94dbc4823fea7d72e7515e1ea2c6480bff77\": container with ID starting with 163408920e5dfddc34b3cd4ccb8c94dbc4823fea7d72e7515e1ea2c6480bff77 not found: ID does not exist"
Jul 14 15:54:21.576323 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.576301    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"]
Jul 14 15:54:21.578576 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:21.578555    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9cd0814a-openshift-default-5c89f9f7bc-w4swt"]
Jul 14 15:54:22.216661 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:22.216631    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b96ac2a-0cd1-4753-8135-8de83a2a1de4" path="/var/lib/kubelet/pods/3b96ac2a-0cd1-4753-8135-8de83a2a1de4/volumes"
Jul 14 15:54:36.359297 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.359268    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"]
Jul 14 15:54:36.359760 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.359690    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b96ac2a-0cd1-4753-8135-8de83a2a1de4" containerName="istio-proxy"
Jul 14 15:54:36.359760 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.359702    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b96ac2a-0cd1-4753-8135-8de83a2a1de4" containerName="istio-proxy"
Jul 14 15:54:36.359760 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.359712    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b96ac2a-0cd1-4753-8135-8de83a2a1de4" containerName="istio-proxy"
Jul 14 15:54:36.359760 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.359718    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b96ac2a-0cd1-4753-8135-8de83a2a1de4" containerName="istio-proxy"
Jul 14 15:54:36.359760 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.359735    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66b9e4de-5f94-4ff6-bfa5-9abe071cc174" containerName="revoke-keys"
Jul 14 15:54:36.359760 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.359741    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="66b9e4de-5f94-4ff6-bfa5-9abe071cc174" containerName="revoke-keys"
Jul 14 15:54:36.359760 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.359751    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5932082a-e704-4841-8b65-dd155f385e07" containerName="maas-api"
Jul 14 15:54:36.359760 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.359757    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5932082a-e704-4841-8b65-dd155f385e07" containerName="maas-api"
Jul 14 15:54:36.360007 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.359817    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5932082a-e704-4841-8b65-dd155f385e07" containerName="maas-api"
Jul 14 15:54:36.360007 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.359827    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="66b9e4de-5f94-4ff6-bfa5-9abe071cc174" containerName="revoke-keys"
Jul 14 15:54:36.360007 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.359835    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b96ac2a-0cd1-4753-8135-8de83a2a1de4" containerName="istio-proxy"
Jul 14 15:54:36.361957 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.361942    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.366852 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.366828    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 15:54:36.366988 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.366874    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5e35a406-openshift-default-dockercfg-4d5fg\""
Jul 14 15:54:36.367134 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.367120    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 15:54:36.367318 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.367306    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 15:54:36.380230 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.380199    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"]
Jul 14 15:54:36.464247 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.464211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnfw5\" (UniqueName: \"kubernetes.io/projected/c330fada-07c2-4717-9c8d-f58312947c38-kube-api-access-gnfw5\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.464422 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.464257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c330fada-07c2-4717-9c8d-f58312947c38-istio-podinfo\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.464422 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.464324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-istio-envoy\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.464422 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.464381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-credential-socket\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.464555 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.464435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c330fada-07c2-4717-9c8d-f58312947c38-istiod-ca-cert\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.464555 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.464458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-istio-data\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.464555 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.464486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c330fada-07c2-4717-9c8d-f58312947c38-istio-token\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.464555 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.464530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-workload-certs\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.464685 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.464555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-workload-socket\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.565288 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.565246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c330fada-07c2-4717-9c8d-f58312947c38-istiod-ca-cert\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.565288 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.565291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-istio-data\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.565570 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.565332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c330fada-07c2-4717-9c8d-f58312947c38-istio-token\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.565570 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.565356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-workload-certs\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.565570 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.565385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-workload-socket\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.565570 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.565414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfw5\" (UniqueName: \"kubernetes.io/projected/c330fada-07c2-4717-9c8d-f58312947c38-kube-api-access-gnfw5\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.565570 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.565457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c330fada-07c2-4717-9c8d-f58312947c38-istio-podinfo\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.565570 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.565528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-istio-envoy\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.565570 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.565566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-credential-socket\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.565934 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.565782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-workload-certs\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.565934 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.565896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-workload-socket\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.565934 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.565917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-istio-data\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.566084 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.566012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-credential-socket\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.566121 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.566087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c330fada-07c2-4717-9c8d-f58312947c38-istiod-ca-cert\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.568113 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.568092    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-istio-envoy\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.568232 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.568194    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c330fada-07c2-4717-9c8d-f58312947c38-istio-podinfo\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.577695 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.577670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfw5\" (UniqueName: \"kubernetes.io/projected/c330fada-07c2-4717-9c8d-f58312947c38-kube-api-access-gnfw5\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.578564 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.578546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c330fada-07c2-4717-9c8d-f58312947c38-istio-token\") pod \"e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") " pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.672972 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.672945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:36.830973 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.830948    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"]
Jul 14 15:54:36.831931 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:54:36.831911    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc330fada_07c2_4717_9c8d_f58312947c38.slice/crio-2ecf616aed4bfaba0b1cdf531f02d4da9e798af137a31932c053987500d67cbf WatchSource:0}: Error finding container 2ecf616aed4bfaba0b1cdf531f02d4da9e798af137a31932c053987500d67cbf: Status 404 returned error can't find the container with id 2ecf616aed4bfaba0b1cdf531f02d4da9e798af137a31932c053987500d67cbf
Jul 14 15:54:36.834024 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.833988    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:54:36.834126 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.834075    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:54:36.834126 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:36.834117    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:54:37.640549 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:37.640498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh" event={"ID":"c330fada-07c2-4717-9c8d-f58312947c38","Type":"ContainerStarted","Data":"7272a6c05d6cf3f39c0d44138b7f7638559a983ce71f4d6e5c6d25258fbff802"}
Jul 14 15:54:37.640952 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:37.640575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh" event={"ID":"c330fada-07c2-4717-9c8d-f58312947c38","Type":"ContainerStarted","Data":"2ecf616aed4bfaba0b1cdf531f02d4da9e798af137a31932c053987500d67cbf"}
Jul 14 15:54:37.662418 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:37.662370    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh" podStartSLOduration=1.662352526 podStartE2EDuration="1.662352526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:54:37.660490396 +0000 UTC m=+2002.001756327" watchObservedRunningTime="2026-07-14 15:54:37.662352526 +0000 UTC m=+2002.003618438"
Jul 14 15:54:37.673620 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:37.673592    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:37.678421 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:37.678400    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:38.646156 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:38.646124    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:38.647152 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:38.647134    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:39.800056 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:39.800024    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"]
Jul 14 15:54:39.800820 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:39.800799    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b96ac2a-0cd1-4753-8135-8de83a2a1de4" containerName="istio-proxy"
Jul 14 15:54:39.803531 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:39.803484    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:54:39.806531 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:39.806492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5e35a406-serving-cert\""
Jul 14 15:54:39.806666 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:39.806567    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 15:54:39.807816 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:39.807798    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 15:54:39.807907 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:39.807803    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npnj7\""
Jul 14 15:54:39.816931 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:39.816912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"]
Jul 14 15:54:39.899185 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:39.899152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-openshift-service-ca\") pod \"maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt\" (UID: \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:54:39.899360 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:39.899194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v4ht\" (UniqueName: \"kubernetes.io/projected/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-kube-api-access-6v4ht\") pod \"maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt\" (UID: \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:54:39.899360 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:39.899231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-maas-api-tls\") pod \"maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt\" (UID: \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:54:40.000678 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:40.000640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-openshift-service-ca\") pod \"maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt\" (UID: \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:54:40.000847 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:40.000692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v4ht\" (UniqueName: \"kubernetes.io/projected/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-kube-api-access-6v4ht\") pod \"maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt\" (UID: \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:54:40.000847 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:40.000723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-maas-api-tls\") pod \"maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt\" (UID: \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:54:40.001269 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:40.001245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-openshift-service-ca\") pod \"maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt\" (UID: \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:54:40.003354 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:40.003332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-maas-api-tls\") pod \"maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt\" (UID: \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:54:40.011715 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:40.011692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v4ht\" (UniqueName: \"kubernetes.io/projected/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-kube-api-access-6v4ht\") pod \"maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt\" (UID: \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:54:40.115187 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:40.115107    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:54:40.262258 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:40.262230    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"]
Jul 14 15:54:40.263573 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:54:40.263549    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f01a00_c41a_40ff_a5ca_93aead4bbf49.slice/crio-bbcd2fac1ca7142092d2f55eee2312ce48c179115660d3bce656f69b5ef9b090 WatchSource:0}: Error finding container bbcd2fac1ca7142092d2f55eee2312ce48c179115660d3bce656f69b5ef9b090: Status 404 returned error can't find the container with id bbcd2fac1ca7142092d2f55eee2312ce48c179115660d3bce656f69b5ef9b090
Jul 14 15:54:40.657527 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:40.657479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt" event={"ID":"a3f01a00-c41a-40ff-a5ca-93aead4bbf49","Type":"ContainerStarted","Data":"bbcd2fac1ca7142092d2f55eee2312ce48c179115660d3bce656f69b5ef9b090"}
Jul 14 15:54:41.664883 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:41.664845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt" event={"ID":"a3f01a00-c41a-40ff-a5ca-93aead4bbf49","Type":"ContainerStarted","Data":"f9819a1259e1df1e0cbadab0ccd6a2eed4aa6ee5ccb4007d23045a1d5a0b3129"}
Jul 14 15:54:41.665368 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:41.664925    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:54:41.683296 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:41.683249    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt" podStartSLOduration=2.304913472 podStartE2EDuration="2.683233593s" totalImagesPullingTime="378.320121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:54:41.682236379 +0000 UTC m=+2006.023502315" watchObservedRunningTime="2026-07-14 15:54:41.683233593 +0000 UTC m=+2006.024499499"
Jul 14 15:54:47.677310 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:47.677278    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:54:54.688692 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:54.688645    2563 patch_prober.go:28] interesting pod/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 14 15:54:54.689097 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:54.688712    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh" podUID="c330fada-07c2-4717-9c8d-f58312947c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 14 15:54:55.748284 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:55.748247    2563 generic.go:356] "Generic (PLEG): container finished" podID="c330fada-07c2-4717-9c8d-f58312947c38" containerID="7272a6c05d6cf3f39c0d44138b7f7638559a983ce71f4d6e5c6d25258fbff802" exitCode=0
Jul 14 15:54:55.748741 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:55.748320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh" event={"ID":"c330fada-07c2-4717-9c8d-f58312947c38","Type":"ContainerDied","Data":"7272a6c05d6cf3f39c0d44138b7f7638559a983ce71f4d6e5c6d25258fbff802"}
Jul 14 15:54:55.748741 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:55.748730    2563 scope.go:117] "RemoveContainer" containerID="7272a6c05d6cf3f39c0d44138b7f7638559a983ce71f4d6e5c6d25258fbff802"
Jul 14 15:54:55.749844 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:55.749813    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:54:55.749932 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:55.749899    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:54:55.749971 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:55.749934    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:54:56.756164 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:56.756126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh" event={"ID":"c330fada-07c2-4717-9c8d-f58312947c38","Type":"ContainerStarted","Data":"97307edffb5d2169adcd1cb9cb0985d995312c53b7e6547dee79068fd9427c91"}
Jul 14 15:54:57.673579 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:57.673531    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:54:57.673771 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:57.673672    2563 patch_prober.go:28] interesting pod/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 14 15:54:57.673771 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:57.673723    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh" podUID="c330fada-07c2-4717-9c8d-f58312947c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 14 15:54:58.674335 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:58.674289    2563 patch_prober.go:28] interesting pod/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 14 15:54:58.674865 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:58.674368    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh" podUID="c330fada-07c2-4717-9c8d-f58312947c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 14 15:54:59.673516 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:59.673476    2563 patch_prober.go:28] interesting pod/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 14 15:54:59.673693 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:54:59.673563    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh" podUID="c330fada-07c2-4717-9c8d-f58312947c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 14 15:55:00.676956 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:00.676927    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:55:00.677323 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:00.677307    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:55:00.678056 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:00.678039    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:55:38.063712 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:38.063632    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz"]
Jul 14 15:55:38.066301 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:38.066282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz"
Jul 14 15:55:38.069006 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:38.068986    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\""
Jul 14 15:55:38.074737 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:38.074714    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz"]
Jul 14 15:55:38.158227 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:38.158191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46b2efc-c45f-46bf-9a9e-2caa17acb588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz\" (UID: \"b46b2efc-c45f-46bf-9a9e-2caa17acb588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz"
Jul 14 15:55:38.259098 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:38.259047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46b2efc-c45f-46bf-9a9e-2caa17acb588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz\" (UID: \"b46b2efc-c45f-46bf-9a9e-2caa17acb588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz"
Jul 14 15:55:38.259701 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:38.259681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46b2efc-c45f-46bf-9a9e-2caa17acb588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz\" (UID: \"b46b2efc-c45f-46bf-9a9e-2caa17acb588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz"
Jul 14 15:55:38.379777 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:38.379692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz"
Jul 14 15:55:38.513600 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:38.513543    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz"]
Jul 14 15:55:38.516157 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:55:38.516128    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46b2efc_c45f_46bf_9a9e_2caa17acb588.slice/crio-78f0d696c26be7c1587886e18c4d4f46715657343c6d7326660f5c5ef98c83ab WatchSource:0}: Error finding container 78f0d696c26be7c1587886e18c4d4f46715657343c6d7326660f5c5ef98c83ab: Status 404 returned error can't find the container with id 78f0d696c26be7c1587886e18c4d4f46715657343c6d7326660f5c5ef98c83ab
Jul 14 15:55:38.992348 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:38.992247    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz" event={"ID":"b46b2efc-c45f-46bf-9a9e-2caa17acb588","Type":"ContainerStarted","Data":"3ad0ef2e50af332781142a8c8c37c1f685438c07b349215459ac6664d56f9568"}
Jul 14 15:55:38.992348 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:38.992307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz" event={"ID":"b46b2efc-c45f-46bf-9a9e-2caa17acb588","Type":"ContainerStarted","Data":"78f0d696c26be7c1587886e18c4d4f46715657343c6d7326660f5c5ef98c83ab"}
Jul 14 15:55:39.010870 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:39.010821    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz" podStartSLOduration=1.010806599 podStartE2EDuration="1.010806599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:55:39.008009468 +0000 UTC m=+2063.349275391" watchObservedRunningTime="2026-07-14 15:55:39.010806599 +0000 UTC m=+2063.352072505"
Jul 14 15:55:39.999482 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:39.999446    2563 generic.go:356] "Generic (PLEG): container finished" podID="b46b2efc-c45f-46bf-9a9e-2caa17acb588" containerID="3ad0ef2e50af332781142a8c8c37c1f685438c07b349215459ac6664d56f9568" exitCode=0
Jul 14 15:55:39.999884 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:39.999517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz" event={"ID":"b46b2efc-c45f-46bf-9a9e-2caa17acb588","Type":"ContainerDied","Data":"3ad0ef2e50af332781142a8c8c37c1f685438c07b349215459ac6664d56f9568"}
Jul 14 15:55:41.145609 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:41.145583    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz"
Jul 14 15:55:41.187840 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:41.187817    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46b2efc-c45f-46bf-9a9e-2caa17acb588-maas-api-service-ca\") pod \"b46b2efc-c45f-46bf-9a9e-2caa17acb588\" (UID: \"b46b2efc-c45f-46bf-9a9e-2caa17acb588\") "
Jul 14 15:55:41.188139 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:41.188117    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46b2efc-c45f-46bf-9a9e-2caa17acb588-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b46b2efc-c45f-46bf-9a9e-2caa17acb588" (UID: "b46b2efc-c45f-46bf-9a9e-2caa17acb588"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:55:41.289943 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:41.289842    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46b2efc-c45f-46bf-9a9e-2caa17acb588-maas-api-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:55:42.018797 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:42.018769    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz"
Jul 14 15:55:42.018797 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:42.018775    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz" event={"ID":"b46b2efc-c45f-46bf-9a9e-2caa17acb588","Type":"ContainerDied","Data":"78f0d696c26be7c1587886e18c4d4f46715657343c6d7326660f5c5ef98c83ab"}
Jul 14 15:55:42.019004 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:42.018820    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78f0d696c26be7c1587886e18c4d4f46715657343c6d7326660f5c5ef98c83ab"
Jul 14 15:55:48.613895 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:48.613860    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"]
Jul 14 15:55:48.614374 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:48.614063    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt" podUID="a3f01a00-c41a-40ff-a5ca-93aead4bbf49" containerName="maas-api" containerID="cri-o://f9819a1259e1df1e0cbadab0ccd6a2eed4aa6ee5ccb4007d23045a1d5a0b3129" gracePeriod=30
Jul 14 15:55:48.873206 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:48.873143    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:55:48.966957 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:48.966925    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-maas-api-tls\") pod \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\" (UID: \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\") "
Jul 14 15:55:48.967167 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:48.967017    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-openshift-service-ca\") pod \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\" (UID: \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\") "
Jul 14 15:55:48.967167 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:48.967104    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v4ht\" (UniqueName: \"kubernetes.io/projected/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-kube-api-access-6v4ht\") pod \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\" (UID: \"a3f01a00-c41a-40ff-a5ca-93aead4bbf49\") "
Jul 14 15:55:48.967363 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:48.967336    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a3f01a00-c41a-40ff-a5ca-93aead4bbf49" (UID: "a3f01a00-c41a-40ff-a5ca-93aead4bbf49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:55:48.967418 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:48.967404    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-openshift-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:55:48.969302 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:48.969280    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-kube-api-access-6v4ht" (OuterVolumeSpecName: "kube-api-access-6v4ht") pod "a3f01a00-c41a-40ff-a5ca-93aead4bbf49" (UID: "a3f01a00-c41a-40ff-a5ca-93aead4bbf49"). InnerVolumeSpecName "kube-api-access-6v4ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:55:48.969407 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:48.969306    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a3f01a00-c41a-40ff-a5ca-93aead4bbf49" (UID: "a3f01a00-c41a-40ff-a5ca-93aead4bbf49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:55:49.059565 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:49.059495    2563 generic.go:356] "Generic (PLEG): container finished" podID="a3f01a00-c41a-40ff-a5ca-93aead4bbf49" containerID="f9819a1259e1df1e0cbadab0ccd6a2eed4aa6ee5ccb4007d23045a1d5a0b3129" exitCode=0
Jul 14 15:55:49.059565 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:49.059547    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt" event={"ID":"a3f01a00-c41a-40ff-a5ca-93aead4bbf49","Type":"ContainerDied","Data":"f9819a1259e1df1e0cbadab0ccd6a2eed4aa6ee5ccb4007d23045a1d5a0b3129"}
Jul 14 15:55:49.059791 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:49.059606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt" event={"ID":"a3f01a00-c41a-40ff-a5ca-93aead4bbf49","Type":"ContainerDied","Data":"bbcd2fac1ca7142092d2f55eee2312ce48c179115660d3bce656f69b5ef9b090"}
Jul 14 15:55:49.059791 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:49.059561    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"
Jul 14 15:55:49.059791 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:49.059628    2563 scope.go:117] "RemoveContainer" containerID="f9819a1259e1df1e0cbadab0ccd6a2eed4aa6ee5ccb4007d23045a1d5a0b3129"
Jul 14 15:55:49.068099 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:49.068075    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v4ht\" (UniqueName: \"kubernetes.io/projected/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-kube-api-access-6v4ht\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:55:49.068234 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:49.068169    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f01a00-c41a-40ff-a5ca-93aead4bbf49-maas-api-tls\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:55:49.068994 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:49.068976    2563 scope.go:117] "RemoveContainer" containerID="f9819a1259e1df1e0cbadab0ccd6a2eed4aa6ee5ccb4007d23045a1d5a0b3129"
Jul 14 15:55:49.069256 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:55:49.069240    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9819a1259e1df1e0cbadab0ccd6a2eed4aa6ee5ccb4007d23045a1d5a0b3129\": container with ID starting with f9819a1259e1df1e0cbadab0ccd6a2eed4aa6ee5ccb4007d23045a1d5a0b3129 not found: ID does not exist" containerID="f9819a1259e1df1e0cbadab0ccd6a2eed4aa6ee5ccb4007d23045a1d5a0b3129"
Jul 14 15:55:49.069306 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:49.069263    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9819a1259e1df1e0cbadab0ccd6a2eed4aa6ee5ccb4007d23045a1d5a0b3129"} err="failed to get container status \"f9819a1259e1df1e0cbadab0ccd6a2eed4aa6ee5ccb4007d23045a1d5a0b3129\": rpc error: code = NotFound desc = could not find container \"f9819a1259e1df1e0cbadab0ccd6a2eed4aa6ee5ccb4007d23045a1d5a0b3129\": container with ID starting with f9819a1259e1df1e0cbadab0ccd6a2eed4aa6ee5ccb4007d23045a1d5a0b3129 not found: ID does not exist"
Jul 14 15:55:49.083644 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:49.083619    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"]
Jul 14 15:55:49.091737 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:49.091714    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e35a406-8b947d8bb-qvrbt"]
Jul 14 15:55:50.216919 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:55:50.216887    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3f01a00-c41a-40ff-a5ca-93aead4bbf49" path="/var/lib/kubelet/pods/a3f01a00-c41a-40ff-a5ca-93aead4bbf49/volumes"
Jul 14 15:56:16.251813 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:16.251786    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:56:16.255984 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:16.255966    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 15:56:16.379301 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:16.379270    2563 scope.go:117] "RemoveContainer" containerID="aaf2bc138371173004fdf90a155577679e0c4d9e785904b9dcb96d4ef37fd33d"
Jul 14 15:56:45.493054 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:45.492975    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"]
Jul 14 15:56:45.493633 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:45.493220    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh" podUID="c330fada-07c2-4717-9c8d-f58312947c38" containerName="istio-proxy" containerID="cri-o://97307edffb5d2169adcd1cb9cb0985d995312c53b7e6547dee79068fd9427c91" gracePeriod=30
Jul 14 15:56:45.679617 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:45.679578    2563 patch_prober.go:28] interesting pod/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 15:56:45.679806 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:45.679657    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh" podUID="c330fada-07c2-4717-9c8d-f58312947c38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 15:56:50.751566 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.751540    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:56:50.877603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.877485    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c330fada-07c2-4717-9c8d-f58312947c38-istio-token\") pod \"c330fada-07c2-4717-9c8d-f58312947c38\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") "
Jul 14 15:56:50.877603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.877553    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c330fada-07c2-4717-9c8d-f58312947c38-istio-podinfo\") pod \"c330fada-07c2-4717-9c8d-f58312947c38\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") "
Jul 14 15:56:50.877603 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.877584    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-workload-certs\") pod \"c330fada-07c2-4717-9c8d-f58312947c38\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") "
Jul 14 15:56:50.877889 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.877615    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-credential-socket\") pod \"c330fada-07c2-4717-9c8d-f58312947c38\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") "
Jul 14 15:56:50.877889 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.877636    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-istio-envoy\") pod \"c330fada-07c2-4717-9c8d-f58312947c38\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") "
Jul 14 15:56:50.877889 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.877672    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnfw5\" (UniqueName: \"kubernetes.io/projected/c330fada-07c2-4717-9c8d-f58312947c38-kube-api-access-gnfw5\") pod \"c330fada-07c2-4717-9c8d-f58312947c38\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") "
Jul 14 15:56:50.877889 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.877714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-workload-socket\") pod \"c330fada-07c2-4717-9c8d-f58312947c38\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") "
Jul 14 15:56:50.877889 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.877791    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-istio-data\") pod \"c330fada-07c2-4717-9c8d-f58312947c38\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") "
Jul 14 15:56:50.877889 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.877824    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c330fada-07c2-4717-9c8d-f58312947c38-istiod-ca-cert\") pod \"c330fada-07c2-4717-9c8d-f58312947c38\" (UID: \"c330fada-07c2-4717-9c8d-f58312947c38\") "
Jul 14 15:56:50.878203 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.877916    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c330fada-07c2-4717-9c8d-f58312947c38" (UID: "c330fada-07c2-4717-9c8d-f58312947c38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:56:50.878203 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.877952    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c330fada-07c2-4717-9c8d-f58312947c38" (UID: "c330fada-07c2-4717-9c8d-f58312947c38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:56:50.878203 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.878159    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c330fada-07c2-4717-9c8d-f58312947c38" (UID: "c330fada-07c2-4717-9c8d-f58312947c38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:56:50.878339 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.878240    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-workload-certs\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:56:50.878454 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.878345    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-credential-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:56:50.878966 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.878941    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c330fada-07c2-4717-9c8d-f58312947c38-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c330fada-07c2-4717-9c8d-f58312947c38" (UID: "c330fada-07c2-4717-9c8d-f58312947c38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:56:50.879313 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.879280    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-istio-data" (OuterVolumeSpecName: "istio-data") pod "c330fada-07c2-4717-9c8d-f58312947c38" (UID: "c330fada-07c2-4717-9c8d-f58312947c38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:56:50.880002 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.879975    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c330fada-07c2-4717-9c8d-f58312947c38-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c330fada-07c2-4717-9c8d-f58312947c38" (UID: "c330fada-07c2-4717-9c8d-f58312947c38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 15:56:50.880093 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.880021    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c330fada-07c2-4717-9c8d-f58312947c38-istio-token" (OuterVolumeSpecName: "istio-token") pod "c330fada-07c2-4717-9c8d-f58312947c38" (UID: "c330fada-07c2-4717-9c8d-f58312947c38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:56:50.880093 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.880045    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c330fada-07c2-4717-9c8d-f58312947c38" (UID: "c330fada-07c2-4717-9c8d-f58312947c38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:56:50.880193 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.880177    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c330fada-07c2-4717-9c8d-f58312947c38-kube-api-access-gnfw5" (OuterVolumeSpecName: "kube-api-access-gnfw5") pod "c330fada-07c2-4717-9c8d-f58312947c38" (UID: "c330fada-07c2-4717-9c8d-f58312947c38"). InnerVolumeSpecName "kube-api-access-gnfw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:56:50.979150 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.979115    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-istio-data\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:56:50.979150 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.979142    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c330fada-07c2-4717-9c8d-f58312947c38-istiod-ca-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:56:50.979150 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.979152    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c330fada-07c2-4717-9c8d-f58312947c38-istio-token\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:56:50.979150 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.979160    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c330fada-07c2-4717-9c8d-f58312947c38-istio-podinfo\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:56:50.979419 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.979168    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-istio-envoy\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:56:50.979419 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.979177    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnfw5\" (UniqueName: \"kubernetes.io/projected/c330fada-07c2-4717-9c8d-f58312947c38-kube-api-access-gnfw5\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:56:50.979419 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:50.979187    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c330fada-07c2-4717-9c8d-f58312947c38-workload-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:56:51.400026 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:51.399991    2563 generic.go:356] "Generic (PLEG): container finished" podID="c330fada-07c2-4717-9c8d-f58312947c38" containerID="97307edffb5d2169adcd1cb9cb0985d995312c53b7e6547dee79068fd9427c91" exitCode=0
Jul 14 15:56:51.400195 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:51.400042    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"
Jul 14 15:56:51.400195 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:51.400084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh" event={"ID":"c330fada-07c2-4717-9c8d-f58312947c38","Type":"ContainerDied","Data":"97307edffb5d2169adcd1cb9cb0985d995312c53b7e6547dee79068fd9427c91"}
Jul 14 15:56:51.400195 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:51.400140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh" event={"ID":"c330fada-07c2-4717-9c8d-f58312947c38","Type":"ContainerDied","Data":"2ecf616aed4bfaba0b1cdf531f02d4da9e798af137a31932c053987500d67cbf"}
Jul 14 15:56:51.400195 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:51.400169    2563 scope.go:117] "RemoveContainer" containerID="97307edffb5d2169adcd1cb9cb0985d995312c53b7e6547dee79068fd9427c91"
Jul 14 15:56:51.410161 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:51.410145    2563 scope.go:117] "RemoveContainer" containerID="7272a6c05d6cf3f39c0d44138b7f7638559a983ce71f4d6e5c6d25258fbff802"
Jul 14 15:56:51.418279 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:51.418261    2563 scope.go:117] "RemoveContainer" containerID="97307edffb5d2169adcd1cb9cb0985d995312c53b7e6547dee79068fd9427c91"
Jul 14 15:56:51.418659 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:56:51.418634    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97307edffb5d2169adcd1cb9cb0985d995312c53b7e6547dee79068fd9427c91\": container with ID starting with 97307edffb5d2169adcd1cb9cb0985d995312c53b7e6547dee79068fd9427c91 not found: ID does not exist" containerID="97307edffb5d2169adcd1cb9cb0985d995312c53b7e6547dee79068fd9427c91"
Jul 14 15:56:51.418734 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:51.418669    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97307edffb5d2169adcd1cb9cb0985d995312c53b7e6547dee79068fd9427c91"} err="failed to get container status \"97307edffb5d2169adcd1cb9cb0985d995312c53b7e6547dee79068fd9427c91\": rpc error: code = NotFound desc = could not find container \"97307edffb5d2169adcd1cb9cb0985d995312c53b7e6547dee79068fd9427c91\": container with ID starting with 97307edffb5d2169adcd1cb9cb0985d995312c53b7e6547dee79068fd9427c91 not found: ID does not exist"
Jul 14 15:56:51.418734 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:51.418688    2563 scope.go:117] "RemoveContainer" containerID="7272a6c05d6cf3f39c0d44138b7f7638559a983ce71f4d6e5c6d25258fbff802"
Jul 14 15:56:51.418925 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:56:51.418909    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7272a6c05d6cf3f39c0d44138b7f7638559a983ce71f4d6e5c6d25258fbff802\": container with ID starting with 7272a6c05d6cf3f39c0d44138b7f7638559a983ce71f4d6e5c6d25258fbff802 not found: ID does not exist" containerID="7272a6c05d6cf3f39c0d44138b7f7638559a983ce71f4d6e5c6d25258fbff802"
Jul 14 15:56:51.418970 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:51.418934    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7272a6c05d6cf3f39c0d44138b7f7638559a983ce71f4d6e5c6d25258fbff802"} err="failed to get container status \"7272a6c05d6cf3f39c0d44138b7f7638559a983ce71f4d6e5c6d25258fbff802\": rpc error: code = NotFound desc = could not find container \"7272a6c05d6cf3f39c0d44138b7f7638559a983ce71f4d6e5c6d25258fbff802\": container with ID starting with 7272a6c05d6cf3f39c0d44138b7f7638559a983ce71f4d6e5c6d25258fbff802 not found: ID does not exist"
Jul 14 15:56:51.428939 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:51.428915    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"]
Jul 14 15:56:51.433407 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:51.433380    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5e35a406-openshift-default-77776b6d6c-5qzvh"]
Jul 14 15:56:52.216227 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:56:52.216194    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c330fada-07c2-4717-9c8d-f58312947c38" path="/var/lib/kubelet/pods/c330fada-07c2-4717-9c8d-f58312947c38/volumes"
Jul 14 15:57:16.404396 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:16.404314    2563 scope.go:117] "RemoveContainer" containerID="33bbc96d243cd0ac9449e040cdb39495439247c3a6435ec1d08e701228835c15"
Jul 14 15:57:41.426088 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.426056    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"]
Jul 14 15:57:41.426678 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.426661    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c330fada-07c2-4717-9c8d-f58312947c38" containerName="istio-proxy"
Jul 14 15:57:41.426728 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.426679    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c330fada-07c2-4717-9c8d-f58312947c38" containerName="istio-proxy"
Jul 14 15:57:41.426728 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.426696    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b46b2efc-c45f-46bf-9a9e-2caa17acb588" containerName="revoke-keys"
Jul 14 15:57:41.426728 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.426702    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b46b2efc-c45f-46bf-9a9e-2caa17acb588" containerName="revoke-keys"
Jul 14 15:57:41.426728 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.426711    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c330fada-07c2-4717-9c8d-f58312947c38" containerName="istio-proxy"
Jul 14 15:57:41.426728 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.426717    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c330fada-07c2-4717-9c8d-f58312947c38" containerName="istio-proxy"
Jul 14 15:57:41.426883 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.426734    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f01a00-c41a-40ff-a5ca-93aead4bbf49" containerName="maas-api"
Jul 14 15:57:41.426883 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.426739    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f01a00-c41a-40ff-a5ca-93aead4bbf49" containerName="maas-api"
Jul 14 15:57:41.426883 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.426799    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b46b2efc-c45f-46bf-9a9e-2caa17acb588" containerName="revoke-keys"
Jul 14 15:57:41.426883 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.426810    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c330fada-07c2-4717-9c8d-f58312947c38" containerName="istio-proxy"
Jul 14 15:57:41.426883 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.426817    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c330fada-07c2-4717-9c8d-f58312947c38" containerName="istio-proxy"
Jul 14 15:57:41.426883 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.426823    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f01a00-c41a-40ff-a5ca-93aead4bbf49" containerName="maas-api"
Jul 14 15:57:41.428722 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.428708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.431720 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.431699    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-eb3623-openshift-default-dockercfg-5qrpz\""
Jul 14 15:57:41.431851 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.431766    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 15:57:41.431851 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.431810    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 15:57:41.431851 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.431819    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 15:57:41.439495 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.439473    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"]
Jul 14 15:57:41.461093 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.461053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istiod-ca-cert\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.461276 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.461118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-envoy\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.461276 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.461153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w85d\" (UniqueName: \"kubernetes.io/projected/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-kube-api-access-5w85d\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.461276 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.461194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-credential-socket\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.461276 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.461220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-podinfo\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.461493 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.461293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-workload-certs\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.461493 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.461362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-workload-socket\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.461493 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.461390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-token\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.461493 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.461449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-data\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.562260 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.562224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-data\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.562260 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.562264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istiod-ca-cert\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.562484 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.562295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-envoy\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.562484 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.562321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w85d\" (UniqueName: \"kubernetes.io/projected/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-kube-api-access-5w85d\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.562484 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.562362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-credential-socket\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.562484 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.562386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-podinfo\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.562484 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.562453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-workload-certs\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.562771 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.562625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-workload-socket\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.562771 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.562668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-token\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.562771 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.562710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-data\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.562930 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.562803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-credential-socket\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.562930 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.562811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-workload-certs\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.563040 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.563016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-workload-socket\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.563097 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.563048    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istiod-ca-cert\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.564778 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.564754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-envoy\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.565126 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.565105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-podinfo\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.571573 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.571471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w85d\" (UniqueName: \"kubernetes.io/projected/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-kube-api-access-5w85d\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.571681 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.571662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-token\") pod \"e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") " pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.740359 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.740264    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:41.882923 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.882901    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"]
Jul 14 15:57:41.884662 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:57:41.884637    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5785718_acd7_43b6_a3f5_49f3a5bc04c1.slice/crio-d57a77a0bc245d3a8fb18a131733616b3027972db28733a8ddbb108aec8f6992 WatchSource:0}: Error finding container d57a77a0bc245d3a8fb18a131733616b3027972db28733a8ddbb108aec8f6992: Status 404 returned error can't find the container with id d57a77a0bc245d3a8fb18a131733616b3027972db28733a8ddbb108aec8f6992
Jul 14 15:57:41.886805 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.886769    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:57:41.886881 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.886838    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:57:41.886881 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:41.886864    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:57:42.667317 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:42.667278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m" event={"ID":"c5785718-acd7-43b6-a3f5-49f3a5bc04c1","Type":"ContainerStarted","Data":"e485a55ab583f17120db38379cc833bffa892d80b428b09a76c136e4e8ae46f1"}
Jul 14 15:57:42.667756 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:42.667339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m" event={"ID":"c5785718-acd7-43b6-a3f5-49f3a5bc04c1","Type":"ContainerStarted","Data":"d57a77a0bc245d3a8fb18a131733616b3027972db28733a8ddbb108aec8f6992"}
Jul 14 15:57:42.696886 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:42.696838    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m" podStartSLOduration=1.696823095 podStartE2EDuration="1.696823095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:57:42.694960189 +0000 UTC m=+2187.036226114" watchObservedRunningTime="2026-07-14 15:57:42.696823095 +0000 UTC m=+2187.038089001"
Jul 14 15:57:42.740692 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:42.740660    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:42.745654 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:42.745631    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:43.673136 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:43.673099    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:43.674105 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:43.674087    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:57:44.787063 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:44.787036    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"]
Jul 14 15:57:44.789685 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:44.789667    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:57:44.792336 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:44.792300    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-eb3623-serving-cert\""
Jul 14 15:57:44.792461 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:44.792437    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 15:57:44.793722 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:44.793687    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npnj7\""
Jul 14 15:57:44.793722 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:44.793700    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 15:57:44.801995 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:44.801975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"]
Jul 14 15:57:44.894953 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:44.894918    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d338fbc2-2c01-48dc-8497-8c3e9284faa4-maas-api-tls\") pod \"maas-api-e2e-inf-a-eb3623-7877474f56-m29lw\" (UID: \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:57:44.895123 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:44.894999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d338fbc2-2c01-48dc-8497-8c3e9284faa4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-eb3623-7877474f56-m29lw\" (UID: \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:57:44.895123 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:44.895016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7rqk\" (UniqueName: \"kubernetes.io/projected/d338fbc2-2c01-48dc-8497-8c3e9284faa4-kube-api-access-g7rqk\") pod \"maas-api-e2e-inf-a-eb3623-7877474f56-m29lw\" (UID: \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:57:44.996560 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:44.996498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d338fbc2-2c01-48dc-8497-8c3e9284faa4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-eb3623-7877474f56-m29lw\" (UID: \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:57:44.996768 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:44.996568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7rqk\" (UniqueName: \"kubernetes.io/projected/d338fbc2-2c01-48dc-8497-8c3e9284faa4-kube-api-access-g7rqk\") pod \"maas-api-e2e-inf-a-eb3623-7877474f56-m29lw\" (UID: \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:57:44.996768 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:44.996705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d338fbc2-2c01-48dc-8497-8c3e9284faa4-maas-api-tls\") pod \"maas-api-e2e-inf-a-eb3623-7877474f56-m29lw\" (UID: \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:57:44.996878 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:57:44.996856    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-serving-cert: secret "maas-api-e2e-inf-a-eb3623-serving-cert" not found
Jul 14 15:57:44.996943 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:57:44.996932    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d338fbc2-2c01-48dc-8497-8c3e9284faa4-maas-api-tls podName:d338fbc2-2c01-48dc-8497-8c3e9284faa4 nodeName:}" failed. No retries permitted until 2026-07-14 15:57:45.49691583 +0000 UTC m=+2189.838181714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d338fbc2-2c01-48dc-8497-8c3e9284faa4-maas-api-tls") pod "maas-api-e2e-inf-a-eb3623-7877474f56-m29lw" (UID: "d338fbc2-2c01-48dc-8497-8c3e9284faa4") : secret "maas-api-e2e-inf-a-eb3623-serving-cert" not found
Jul 14 15:57:44.997211 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:44.997192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d338fbc2-2c01-48dc-8497-8c3e9284faa4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-eb3623-7877474f56-m29lw\" (UID: \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:57:45.009388 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:45.009362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7rqk\" (UniqueName: \"kubernetes.io/projected/d338fbc2-2c01-48dc-8497-8c3e9284faa4-kube-api-access-g7rqk\") pod \"maas-api-e2e-inf-a-eb3623-7877474f56-m29lw\" (UID: \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:57:45.503166 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:45.503135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d338fbc2-2c01-48dc-8497-8c3e9284faa4-maas-api-tls\") pod \"maas-api-e2e-inf-a-eb3623-7877474f56-m29lw\" (UID: \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:57:45.505720 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:45.505692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d338fbc2-2c01-48dc-8497-8c3e9284faa4-maas-api-tls\") pod \"maas-api-e2e-inf-a-eb3623-7877474f56-m29lw\" (UID: \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:57:45.701964 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:45.701932    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:57:45.835095 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:45.835071    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"]
Jul 14 15:57:45.837107 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:57:45.837080    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd338fbc2_2c01_48dc_8497_8c3e9284faa4.slice/crio-e034984a85228660a97643644845b4dd882201b135e74ded1e617750bc559d85 WatchSource:0}: Error finding container e034984a85228660a97643644845b4dd882201b135e74ded1e617750bc559d85: Status 404 returned error can't find the container with id e034984a85228660a97643644845b4dd882201b135e74ded1e617750bc559d85
Jul 14 15:57:45.838913 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:45.838896    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 15:57:46.689180 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:46.689148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw" event={"ID":"d338fbc2-2c01-48dc-8497-8c3e9284faa4","Type":"ContainerStarted","Data":"efb675c47ef0d2d205cf901d9b332d7858766d05389c0919a311d4be32933180"}
Jul 14 15:57:46.689380 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:46.689195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw" event={"ID":"d338fbc2-2c01-48dc-8497-8c3e9284faa4","Type":"ContainerStarted","Data":"e034984a85228660a97643644845b4dd882201b135e74ded1e617750bc559d85"}
Jul 14 15:57:46.689380 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:46.689232    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:57:46.707110 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:46.707068    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw" podStartSLOduration=2.268347064 podStartE2EDuration="2.707052262s" totalImagesPullingTime="438.705198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:57:46.705052735 +0000 UTC m=+2191.046318676" watchObservedRunningTime="2026-07-14 15:57:46.707052262 +0000 UTC m=+2191.048318168"
Jul 14 15:57:49.812682 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:49.812650    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"]
Jul 14 15:57:49.816159 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:49.816143    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:49.819408 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:49.819387    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-81addd-openshift-default-dockercfg-nt4ml\""
Jul 14 15:57:49.827831 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:49.827808    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"]
Jul 14 15:57:49.943515 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:49.943460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-data\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:49.943798 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:49.943766    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-credential-socket\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:49.943910 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:49.943897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-token\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:49.944047 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:49.944035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istiod-ca-cert\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:49.944227 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:49.944214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-envoy\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:49.944330 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:49.944318    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-podinfo\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:49.944423 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:49.944411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-workload-socket\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:49.944541 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:49.944527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt5c9\" (UniqueName: \"kubernetes.io/projected/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-kube-api-access-bt5c9\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:49.944647 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:49.944635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-workload-certs\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.045926 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.045887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-token\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.046099 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.045954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istiod-ca-cert\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.046099 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.046017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-envoy\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.046099 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.046039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-podinfo\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.046099 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.046069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-workload-socket\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.046284 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.046097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5c9\" (UniqueName: \"kubernetes.io/projected/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-kube-api-access-bt5c9\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.046284 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.046123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-workload-certs\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.046284 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.046160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-data\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.046450 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.046315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-credential-socket\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.046597 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.046570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-workload-socket\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.046673 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.046641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-workload-certs\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.046811 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.046792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-data\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.046877 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.046838    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istiod-ca-cert\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.047022 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.046999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-credential-socket\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.048579 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.048562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-envoy\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.048636 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.048624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-podinfo\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.055456 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.055434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-token\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.055673 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.055657    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5c9\" (UniqueName: \"kubernetes.io/projected/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-kube-api-access-bt5c9\") pod \"e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") " pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.127584 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.127480    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:50.272051 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.272024    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"]
Jul 14 15:57:50.273322 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:57:50.273292    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11bdbff9_93fa_4a57_9f03_e1f0bb01c792.slice/crio-e37df31ee83cf9471989c296c1df260a695e6a784341af27464349be7509d5ca WatchSource:0}: Error finding container e37df31ee83cf9471989c296c1df260a695e6a784341af27464349be7509d5ca: Status 404 returned error can't find the container with id e37df31ee83cf9471989c296c1df260a695e6a784341af27464349be7509d5ca
Jul 14 15:57:50.275512 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.275470    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:57:50.275587 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.275559    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:57:50.275634 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.275591    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:57:50.713853 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.713820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd" event={"ID":"11bdbff9-93fa-4a57-9f03-e1f0bb01c792","Type":"ContainerStarted","Data":"31ca3c5cbfca599a2a5f7aa99b00a795e95c43cf41bf37275d9553d9b2da25bd"}
Jul 14 15:57:50.714040 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.713872    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd" event={"ID":"11bdbff9-93fa-4a57-9f03-e1f0bb01c792","Type":"ContainerStarted","Data":"e37df31ee83cf9471989c296c1df260a695e6a784341af27464349be7509d5ca"}
Jul 14 15:57:50.733805 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:50.733759    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd" podStartSLOduration=1.73374265 podStartE2EDuration="1.73374265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:57:50.732222131 +0000 UTC m=+2195.073488058" watchObservedRunningTime="2026-07-14 15:57:50.73374265 +0000 UTC m=+2195.075008555"
Jul 14 15:57:51.128424 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:51.128337    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:51.133254 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:51.133232    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:51.718824 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:51.718798    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:51.719846 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:51.719821    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:57:52.700168 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:52.700136    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:57:52.794009 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:52.793978    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"]
Jul 14 15:57:52.797796 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:52.797768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:57:52.800933 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:52.800912    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-81addd-serving-cert\""
Jul 14 15:57:52.810812 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:52.810784    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"]
Jul 14 15:57:52.875018 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:52.874984    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbbnw\" (UniqueName: \"kubernetes.io/projected/224b1ae9-152e-4e2e-ade1-4aafe295559f-kube-api-access-rbbnw\") pod \"maas-api-e2e-inf-b-81addd-797df5db96-6tzb5\" (UID: \"224b1ae9-152e-4e2e-ade1-4aafe295559f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:57:52.875201 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:52.875069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224b1ae9-152e-4e2e-ade1-4aafe295559f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-81addd-797df5db96-6tzb5\" (UID: \"224b1ae9-152e-4e2e-ade1-4aafe295559f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:57:52.875201 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:52.875097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224b1ae9-152e-4e2e-ade1-4aafe295559f-maas-api-tls\") pod \"maas-api-e2e-inf-b-81addd-797df5db96-6tzb5\" (UID: \"224b1ae9-152e-4e2e-ade1-4aafe295559f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:57:52.976577 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:52.976468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbbnw\" (UniqueName: \"kubernetes.io/projected/224b1ae9-152e-4e2e-ade1-4aafe295559f-kube-api-access-rbbnw\") pod \"maas-api-e2e-inf-b-81addd-797df5db96-6tzb5\" (UID: \"224b1ae9-152e-4e2e-ade1-4aafe295559f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:57:52.976741 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:52.976648    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224b1ae9-152e-4e2e-ade1-4aafe295559f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-81addd-797df5db96-6tzb5\" (UID: \"224b1ae9-152e-4e2e-ade1-4aafe295559f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:57:52.976741 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:52.976693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224b1ae9-152e-4e2e-ade1-4aafe295559f-maas-api-tls\") pod \"maas-api-e2e-inf-b-81addd-797df5db96-6tzb5\" (UID: \"224b1ae9-152e-4e2e-ade1-4aafe295559f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:57:52.976865 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:57:52.976846    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-serving-cert: secret "maas-api-e2e-inf-b-81addd-serving-cert" not found
Jul 14 15:57:52.976940 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:57:52.976929    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/224b1ae9-152e-4e2e-ade1-4aafe295559f-maas-api-tls podName:224b1ae9-152e-4e2e-ade1-4aafe295559f nodeName:}" failed. No retries permitted until 2026-07-14 15:57:53.47690796 +0000 UTC m=+2197.818173846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/224b1ae9-152e-4e2e-ade1-4aafe295559f-maas-api-tls") pod "maas-api-e2e-inf-b-81addd-797df5db96-6tzb5" (UID: "224b1ae9-152e-4e2e-ade1-4aafe295559f") : secret "maas-api-e2e-inf-b-81addd-serving-cert" not found
Jul 14 15:57:52.977338 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:52.977316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224b1ae9-152e-4e2e-ade1-4aafe295559f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-81addd-797df5db96-6tzb5\" (UID: \"224b1ae9-152e-4e2e-ade1-4aafe295559f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:57:52.989128 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:52.989104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbbnw\" (UniqueName: \"kubernetes.io/projected/224b1ae9-152e-4e2e-ade1-4aafe295559f-kube-api-access-rbbnw\") pod \"maas-api-e2e-inf-b-81addd-797df5db96-6tzb5\" (UID: \"224b1ae9-152e-4e2e-ade1-4aafe295559f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:57:53.483009 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:53.482958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224b1ae9-152e-4e2e-ade1-4aafe295559f-maas-api-tls\") pod \"maas-api-e2e-inf-b-81addd-797df5db96-6tzb5\" (UID: \"224b1ae9-152e-4e2e-ade1-4aafe295559f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:57:53.486545 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:53.486493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224b1ae9-152e-4e2e-ade1-4aafe295559f-maas-api-tls\") pod \"maas-api-e2e-inf-b-81addd-797df5db96-6tzb5\" (UID: \"224b1ae9-152e-4e2e-ade1-4aafe295559f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:57:53.711454 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:53.711419    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:57:53.898658 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:53.898636    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"]
Jul 14 15:57:53.900362 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:57:53.900336    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224b1ae9_152e_4e2e_ade1_4aafe295559f.slice/crio-a5d6b3cae72abe4636e833ba2394ef07d4038a50c8c62cf930ca52e9687989dd WatchSource:0}: Error finding container a5d6b3cae72abe4636e833ba2394ef07d4038a50c8c62cf930ca52e9687989dd: Status 404 returned error can't find the container with id a5d6b3cae72abe4636e833ba2394ef07d4038a50c8c62cf930ca52e9687989dd
Jul 14 15:57:54.744301 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:54.744255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5" event={"ID":"224b1ae9-152e-4e2e-ade1-4aafe295559f","Type":"ContainerStarted","Data":"3880e47f9d102d7a8da2f4915299db7383fca0672ac0b60af29809eaffc8cbe7"}
Jul 14 15:57:54.744856 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:54.744329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5" event={"ID":"224b1ae9-152e-4e2e-ade1-4aafe295559f","Type":"ContainerStarted","Data":"a5d6b3cae72abe4636e833ba2394ef07d4038a50c8c62cf930ca52e9687989dd"}
Jul 14 15:57:54.745289 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:54.745261    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:57:54.764341 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:54.764276    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5" podStartSLOduration=2.36812112 podStartE2EDuration="2.764258835s" totalImagesPullingTime="396.137715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:57:54.763276855 +0000 UTC m=+2199.104542754" watchObservedRunningTime="2026-07-14 15:57:54.764258835 +0000 UTC m=+2199.105524778"
Jul 14 15:57:59.715685 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:59.715646    2563 patch_prober.go:28] interesting pod/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 14 15:57:59.716068 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:57:59.715716    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m" podUID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 14 15:58:00.785619 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:00.784608    2563 generic.go:356] "Generic (PLEG): container finished" podID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" containerID="e485a55ab583f17120db38379cc833bffa892d80b428b09a76c136e4e8ae46f1" exitCode=0
Jul 14 15:58:00.785619 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:00.784791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m" event={"ID":"c5785718-acd7-43b6-a3f5-49f3a5bc04c1","Type":"ContainerDied","Data":"e485a55ab583f17120db38379cc833bffa892d80b428b09a76c136e4e8ae46f1"}
Jul 14 15:58:00.785619 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:00.785304    2563 scope.go:117] "RemoveContainer" containerID="e485a55ab583f17120db38379cc833bffa892d80b428b09a76c136e4e8ae46f1"
Jul 14 15:58:00.788616 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:00.788248    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:58:00.788616 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:00.788342    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:58:00.788616 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:00.788390    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:58:01.769066 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:01.769024    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:58:01.794092 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:01.794056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m" event={"ID":"c5785718-acd7-43b6-a3f5-49f3a5bc04c1","Type":"ContainerStarted","Data":"7a3ab84c50cfad618585a241684f85ebbc823e552547d0fadf8139e3df3fe229"}
Jul 14 15:58:02.741288 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:02.741246    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:58:02.741530 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:02.741377    2563 patch_prober.go:28] interesting pod/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 14 15:58:02.741530 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:02.741434    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m" podUID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 14 15:58:03.741008 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:03.740969    2563 patch_prober.go:28] interesting pod/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 14 15:58:03.741621 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:03.741588    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m" podUID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 14 15:58:04.741599 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:04.741556    2563 patch_prober.go:28] interesting pod/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 14 15:58:04.742111 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:04.741635    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m" podUID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 14 15:58:05.746181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:05.746152    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:58:05.746600 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:05.746540    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:58:05.747202 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:05.747182    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:58:07.760705 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:07.760660    2563 patch_prober.go:28] interesting pod/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 14 15:58:07.761105 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:07.760731    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd" podUID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 14 15:58:08.834567 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:08.834533    2563 generic.go:356] "Generic (PLEG): container finished" podID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" containerID="31ca3c5cbfca599a2a5f7aa99b00a795e95c43cf41bf37275d9553d9b2da25bd" exitCode=0
Jul 14 15:58:08.834969 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:08.834609    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd" event={"ID":"11bdbff9-93fa-4a57-9f03-e1f0bb01c792","Type":"ContainerDied","Data":"31ca3c5cbfca599a2a5f7aa99b00a795e95c43cf41bf37275d9553d9b2da25bd"}
Jul 14 15:58:08.835023 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:08.835008    2563 scope.go:117] "RemoveContainer" containerID="31ca3c5cbfca599a2a5f7aa99b00a795e95c43cf41bf37275d9553d9b2da25bd"
Jul 14 15:58:08.836090 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:08.836059    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:58:08.836205 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:08.836142    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:58:08.836205 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:08.836189    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 15:58:09.847355 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:09.847311    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd" event={"ID":"11bdbff9-93fa-4a57-9f03-e1f0bb01c792","Type":"ContainerStarted","Data":"d2556619aeb6d00b69135459334fe3ef772eb1ace84d41a88bb2d6d27bfc146c"}
Jul 14 15:58:10.128400 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:10.128228    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:58:10.128604 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:10.128532    2563 patch_prober.go:28] interesting pod/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Jul 14 15:58:10.128675 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:10.128594    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd" podUID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Jul 14 15:58:11.128970 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:11.128931    2563 patch_prober.go:28] interesting pod/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Jul 14 15:58:11.129569 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:11.129533    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd" podUID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Jul 14 15:58:12.128978 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:12.128937    2563 patch_prober.go:28] interesting pod/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Jul 14 15:58:12.129632 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:12.129601    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd" podUID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Jul 14 15:58:13.129002 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:13.128954    2563 patch_prober.go:28] interesting pod/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Jul 14 15:58:13.129525 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:13.129033    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd" podUID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Jul 14 15:58:14.133291 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:14.133252    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:58:14.133700 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:14.133649    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:58:14.134432 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:58:14.134416    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 15:59:07.205733 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:07.205693    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l"]
Jul 14 15:59:07.208421 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:07.208404    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l"
Jul 14 15:59:07.211164 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:07.211141    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\""
Jul 14 15:59:07.216705 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:07.216681    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l"]
Jul 14 15:59:07.321765 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:07.321727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11074954-387e-4b6d-bb73-726d19215daf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eb3623-r455l\" (UID: \"11074954-387e-4b6d-bb73-726d19215daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l"
Jul 14 15:59:07.422398 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:07.422362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11074954-387e-4b6d-bb73-726d19215daf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eb3623-r455l\" (UID: \"11074954-387e-4b6d-bb73-726d19215daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l"
Jul 14 15:59:07.423008 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:07.422984    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11074954-387e-4b6d-bb73-726d19215daf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eb3623-r455l\" (UID: \"11074954-387e-4b6d-bb73-726d19215daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l"
Jul 14 15:59:07.520823 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:07.520725    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l"
Jul 14 15:59:07.664786 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:07.664751    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l"]
Jul 14 15:59:07.665623 ip-10-0-136-210 kubenswrapper[2563]: W0714 15:59:07.665601    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11074954_387e_4b6d_bb73_726d19215daf.slice/crio-7a024465d6eb8d4c92f7dc1fab52160c38f1716180d55af7d40dff9b03eb6003 WatchSource:0}: Error finding container 7a024465d6eb8d4c92f7dc1fab52160c38f1716180d55af7d40dff9b03eb6003: Status 404 returned error can't find the container with id 7a024465d6eb8d4c92f7dc1fab52160c38f1716180d55af7d40dff9b03eb6003
Jul 14 15:59:08.175631 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:08.175590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l" event={"ID":"11074954-387e-4b6d-bb73-726d19215daf","Type":"ContainerStarted","Data":"b96bd819ccac8f39010e692fe01d4ef053bbfdc707612e19b73bcd52b8890762"}
Jul 14 15:59:08.175815 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:08.175655    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l" event={"ID":"11074954-387e-4b6d-bb73-726d19215daf","Type":"ContainerStarted","Data":"7a024465d6eb8d4c92f7dc1fab52160c38f1716180d55af7d40dff9b03eb6003"}
Jul 14 15:59:08.194087 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:08.194034    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l" podStartSLOduration=1.194018643 podStartE2EDuration="1.194018643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:59:08.19193869 +0000 UTC m=+2272.533204600" watchObservedRunningTime="2026-07-14 15:59:08.194018643 +0000 UTC m=+2272.535284550"
Jul 14 15:59:09.183407 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:09.183379    2563 generic.go:356] "Generic (PLEG): container finished" podID="11074954-387e-4b6d-bb73-726d19215daf" containerID="b96bd819ccac8f39010e692fe01d4ef053bbfdc707612e19b73bcd52b8890762" exitCode=0
Jul 14 15:59:09.183822 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:09.183471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l" event={"ID":"11074954-387e-4b6d-bb73-726d19215daf","Type":"ContainerDied","Data":"b96bd819ccac8f39010e692fe01d4ef053bbfdc707612e19b73bcd52b8890762"}
Jul 14 15:59:10.330911 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:10.330888    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l"
Jul 14 15:59:10.454740 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:10.454642    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11074954-387e-4b6d-bb73-726d19215daf-maas-api-service-ca\") pod \"11074954-387e-4b6d-bb73-726d19215daf\" (UID: \"11074954-387e-4b6d-bb73-726d19215daf\") "
Jul 14 15:59:10.455012 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:10.454989    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11074954-387e-4b6d-bb73-726d19215daf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "11074954-387e-4b6d-bb73-726d19215daf" (UID: "11074954-387e-4b6d-bb73-726d19215daf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:59:10.555898 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:10.555856    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11074954-387e-4b6d-bb73-726d19215daf-maas-api-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:11.195296 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:11.195268    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l"
Jul 14 15:59:11.195474 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:11.195292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb3623-r455l" event={"ID":"11074954-387e-4b6d-bb73-726d19215daf","Type":"ContainerDied","Data":"7a024465d6eb8d4c92f7dc1fab52160c38f1716180d55af7d40dff9b03eb6003"}
Jul 14 15:59:11.195474 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:11.195339    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a024465d6eb8d4c92f7dc1fab52160c38f1716180d55af7d40dff9b03eb6003"
Jul 14 15:59:17.678986 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:17.678951    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"]
Jul 14 15:59:17.679646 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:17.679240    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw" podUID="d338fbc2-2c01-48dc-8497-8c3e9284faa4" containerName="maas-api" containerID="cri-o://efb675c47ef0d2d205cf901d9b332d7858766d05389c0919a311d4be32933180" gracePeriod=30
Jul 14 15:59:17.696050 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:17.696005    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw" podUID="d338fbc2-2c01-48dc-8497-8c3e9284faa4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.67:8443/health\": dial tcp 10.134.0.67:8443: connect: connection refused"
Jul 14 15:59:17.937932 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:17.937860    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:59:18.031831 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.031796    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d338fbc2-2c01-48dc-8497-8c3e9284faa4-maas-api-tls\") pod \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\" (UID: \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\") "
Jul 14 15:59:18.032006 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.031853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7rqk\" (UniqueName: \"kubernetes.io/projected/d338fbc2-2c01-48dc-8497-8c3e9284faa4-kube-api-access-g7rqk\") pod \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\" (UID: \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\") "
Jul 14 15:59:18.032006 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.031923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d338fbc2-2c01-48dc-8497-8c3e9284faa4-openshift-service-ca\") pod \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\" (UID: \"d338fbc2-2c01-48dc-8497-8c3e9284faa4\") "
Jul 14 15:59:18.032274 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.032244    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d338fbc2-2c01-48dc-8497-8c3e9284faa4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d338fbc2-2c01-48dc-8497-8c3e9284faa4" (UID: "d338fbc2-2c01-48dc-8497-8c3e9284faa4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:59:18.034076 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.034046    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d338fbc2-2c01-48dc-8497-8c3e9284faa4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d338fbc2-2c01-48dc-8497-8c3e9284faa4" (UID: "d338fbc2-2c01-48dc-8497-8c3e9284faa4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:59:18.034171 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.034095    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d338fbc2-2c01-48dc-8497-8c3e9284faa4-kube-api-access-g7rqk" (OuterVolumeSpecName: "kube-api-access-g7rqk") pod "d338fbc2-2c01-48dc-8497-8c3e9284faa4" (UID: "d338fbc2-2c01-48dc-8497-8c3e9284faa4"). InnerVolumeSpecName "kube-api-access-g7rqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:59:18.132650 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.132611    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d338fbc2-2c01-48dc-8497-8c3e9284faa4-maas-api-tls\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:18.132650 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.132642    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7rqk\" (UniqueName: \"kubernetes.io/projected/d338fbc2-2c01-48dc-8497-8c3e9284faa4-kube-api-access-g7rqk\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:18.132650 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.132652    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d338fbc2-2c01-48dc-8497-8c3e9284faa4-openshift-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:18.240832 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.240800    2563 generic.go:356] "Generic (PLEG): container finished" podID="d338fbc2-2c01-48dc-8497-8c3e9284faa4" containerID="efb675c47ef0d2d205cf901d9b332d7858766d05389c0919a311d4be32933180" exitCode=0
Jul 14 15:59:18.241008 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.240857    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"
Jul 14 15:59:18.241008 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.240867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw" event={"ID":"d338fbc2-2c01-48dc-8497-8c3e9284faa4","Type":"ContainerDied","Data":"efb675c47ef0d2d205cf901d9b332d7858766d05389c0919a311d4be32933180"}
Jul 14 15:59:18.241008 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.240918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw" event={"ID":"d338fbc2-2c01-48dc-8497-8c3e9284faa4","Type":"ContainerDied","Data":"e034984a85228660a97643644845b4dd882201b135e74ded1e617750bc559d85"}
Jul 14 15:59:18.241008 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.240944    2563 scope.go:117] "RemoveContainer" containerID="efb675c47ef0d2d205cf901d9b332d7858766d05389c0919a311d4be32933180"
Jul 14 15:59:18.250708 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.250684    2563 scope.go:117] "RemoveContainer" containerID="efb675c47ef0d2d205cf901d9b332d7858766d05389c0919a311d4be32933180"
Jul 14 15:59:18.250998 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:59:18.250977    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb675c47ef0d2d205cf901d9b332d7858766d05389c0919a311d4be32933180\": container with ID starting with efb675c47ef0d2d205cf901d9b332d7858766d05389c0919a311d4be32933180 not found: ID does not exist" containerID="efb675c47ef0d2d205cf901d9b332d7858766d05389c0919a311d4be32933180"
Jul 14 15:59:18.251070 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.251012    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb675c47ef0d2d205cf901d9b332d7858766d05389c0919a311d4be32933180"} err="failed to get container status \"efb675c47ef0d2d205cf901d9b332d7858766d05389c0919a311d4be32933180\": rpc error: code = NotFound desc = could not find container \"efb675c47ef0d2d205cf901d9b332d7858766d05389c0919a311d4be32933180\": container with ID starting with efb675c47ef0d2d205cf901d9b332d7858766d05389c0919a311d4be32933180 not found: ID does not exist"
Jul 14 15:59:18.264170 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.264142    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"]
Jul 14 15:59:18.269623 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:18.269588    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eb3623-7877474f56-m29lw"]
Jul 14 15:59:20.218339 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:20.218306    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d338fbc2-2c01-48dc-8497-8c3e9284faa4" path="/var/lib/kubelet/pods/d338fbc2-2c01-48dc-8497-8c3e9284faa4/volumes"
Jul 14 15:59:33.010990 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:33.010951    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"]
Jul 14 15:59:33.011405 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:33.011216    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m" podUID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" containerName="istio-proxy" containerID="cri-o://7a3ab84c50cfad618585a241684f85ebbc823e552547d0fadf8139e3df3fe229" gracePeriod=30
Jul 14 15:59:35.748710 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:35.748670    2563 patch_prober.go:28] interesting pod/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 15:59:35.749162 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:35.748748    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m" podUID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 15:59:38.274356 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.274333    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:59:38.323912 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.323815    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-token\") pod \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") "
Jul 14 15:59:38.323912 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.323905    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-podinfo\") pod \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") "
Jul 14 15:59:38.324147 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.323933    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w85d\" (UniqueName: \"kubernetes.io/projected/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-kube-api-access-5w85d\") pod \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") "
Jul 14 15:59:38.324147 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.323968    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-credential-socket\") pod \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") "
Jul 14 15:59:38.324147 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.324116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-data\") pod \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") "
Jul 14 15:59:38.324305 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.324195    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-envoy\") pod \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") "
Jul 14 15:59:38.324305 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.324226    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-workload-socket\") pod \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") "
Jul 14 15:59:38.324305 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.324246    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c5785718-acd7-43b6-a3f5-49f3a5bc04c1" (UID: "c5785718-acd7-43b6-a3f5-49f3a5bc04c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:59:38.324305 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.324299    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istiod-ca-cert\") pod \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") "
Jul 14 15:59:38.324542 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.324328    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-workload-certs\") pod \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\" (UID: \"c5785718-acd7-43b6-a3f5-49f3a5bc04c1\") "
Jul 14 15:59:38.324604 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.324556    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c5785718-acd7-43b6-a3f5-49f3a5bc04c1" (UID: "c5785718-acd7-43b6-a3f5-49f3a5bc04c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:59:38.324829 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.324772    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-credential-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:38.324829 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.324785    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c5785718-acd7-43b6-a3f5-49f3a5bc04c1" (UID: "c5785718-acd7-43b6-a3f5-49f3a5bc04c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:59:38.324829 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.324704    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c5785718-acd7-43b6-a3f5-49f3a5bc04c1" (UID: "c5785718-acd7-43b6-a3f5-49f3a5bc04c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:59:38.324829 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.324796    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-workload-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:38.325552 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.325484    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-data" (OuterVolumeSpecName: "istio-data") pod "c5785718-acd7-43b6-a3f5-49f3a5bc04c1" (UID: "c5785718-acd7-43b6-a3f5-49f3a5bc04c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:59:38.326598 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.326479    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c5785718-acd7-43b6-a3f5-49f3a5bc04c1" (UID: "c5785718-acd7-43b6-a3f5-49f3a5bc04c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 15:59:38.326712 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.326675    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c5785718-acd7-43b6-a3f5-49f3a5bc04c1" (UID: "c5785718-acd7-43b6-a3f5-49f3a5bc04c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:59:38.327012 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.326991    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-token" (OuterVolumeSpecName: "istio-token") pod "c5785718-acd7-43b6-a3f5-49f3a5bc04c1" (UID: "c5785718-acd7-43b6-a3f5-49f3a5bc04c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:59:38.327113 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.327067    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-kube-api-access-5w85d" (OuterVolumeSpecName: "kube-api-access-5w85d") pod "c5785718-acd7-43b6-a3f5-49f3a5bc04c1" (UID: "c5785718-acd7-43b6-a3f5-49f3a5bc04c1"). InnerVolumeSpecName "kube-api-access-5w85d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:59:38.362333 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.362287    2563 generic.go:356] "Generic (PLEG): container finished" podID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" containerID="7a3ab84c50cfad618585a241684f85ebbc823e552547d0fadf8139e3df3fe229" exitCode=0
Jul 14 15:59:38.362519 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.362341    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"
Jul 14 15:59:38.362519 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.362350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m" event={"ID":"c5785718-acd7-43b6-a3f5-49f3a5bc04c1","Type":"ContainerDied","Data":"7a3ab84c50cfad618585a241684f85ebbc823e552547d0fadf8139e3df3fe229"}
Jul 14 15:59:38.362519 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.362417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m" event={"ID":"c5785718-acd7-43b6-a3f5-49f3a5bc04c1","Type":"ContainerDied","Data":"d57a77a0bc245d3a8fb18a131733616b3027972db28733a8ddbb108aec8f6992"}
Jul 14 15:59:38.362519 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.362450    2563 scope.go:117] "RemoveContainer" containerID="7a3ab84c50cfad618585a241684f85ebbc823e552547d0fadf8139e3df3fe229"
Jul 14 15:59:38.372013 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.371992    2563 scope.go:117] "RemoveContainer" containerID="e485a55ab583f17120db38379cc833bffa892d80b428b09a76c136e4e8ae46f1"
Jul 14 15:59:38.380767 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.380745    2563 scope.go:117] "RemoveContainer" containerID="7a3ab84c50cfad618585a241684f85ebbc823e552547d0fadf8139e3df3fe229"
Jul 14 15:59:38.381048 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:59:38.381031    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3ab84c50cfad618585a241684f85ebbc823e552547d0fadf8139e3df3fe229\": container with ID starting with 7a3ab84c50cfad618585a241684f85ebbc823e552547d0fadf8139e3df3fe229 not found: ID does not exist" containerID="7a3ab84c50cfad618585a241684f85ebbc823e552547d0fadf8139e3df3fe229"
Jul 14 15:59:38.381101 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.381056    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3ab84c50cfad618585a241684f85ebbc823e552547d0fadf8139e3df3fe229"} err="failed to get container status \"7a3ab84c50cfad618585a241684f85ebbc823e552547d0fadf8139e3df3fe229\": rpc error: code = NotFound desc = could not find container \"7a3ab84c50cfad618585a241684f85ebbc823e552547d0fadf8139e3df3fe229\": container with ID starting with 7a3ab84c50cfad618585a241684f85ebbc823e552547d0fadf8139e3df3fe229 not found: ID does not exist"
Jul 14 15:59:38.381101 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.381073    2563 scope.go:117] "RemoveContainer" containerID="e485a55ab583f17120db38379cc833bffa892d80b428b09a76c136e4e8ae46f1"
Jul 14 15:59:38.381293 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:59:38.381274    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e485a55ab583f17120db38379cc833bffa892d80b428b09a76c136e4e8ae46f1\": container with ID starting with e485a55ab583f17120db38379cc833bffa892d80b428b09a76c136e4e8ae46f1 not found: ID does not exist" containerID="e485a55ab583f17120db38379cc833bffa892d80b428b09a76c136e4e8ae46f1"
Jul 14 15:59:38.381331 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.381297    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e485a55ab583f17120db38379cc833bffa892d80b428b09a76c136e4e8ae46f1"} err="failed to get container status \"e485a55ab583f17120db38379cc833bffa892d80b428b09a76c136e4e8ae46f1\": rpc error: code = NotFound desc = could not find container \"e485a55ab583f17120db38379cc833bffa892d80b428b09a76c136e4e8ae46f1\": container with ID starting with e485a55ab583f17120db38379cc833bffa892d80b428b09a76c136e4e8ae46f1 not found: ID does not exist"
Jul 14 15:59:38.387641 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.387618    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"]
Jul 14 15:59:38.392705 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.392679    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-eb3623-openshift-default-5b9c6b4d8c-7q49m"]
Jul 14 15:59:38.426181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.426142    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-envoy\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:38.426181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.426175    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istiod-ca-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:38.426181 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.426188    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-workload-certs\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:38.426415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.426199    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-token\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:38.426415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.426212    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-podinfo\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:38.426415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.426225    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w85d\" (UniqueName: \"kubernetes.io/projected/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-kube-api-access-5w85d\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:38.426415 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:38.426237    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5785718-acd7-43b6-a3f5-49f3a5bc04c1-istio-data\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:40.217119 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:40.217090    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" path="/var/lib/kubelet/pods/c5785718-acd7-43b6-a3f5-49f3a5bc04c1/volumes"
Jul 14 15:59:43.858076 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:43.858032    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"]
Jul 14 15:59:43.858583 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:43.858285    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5" podUID="224b1ae9-152e-4e2e-ade1-4aafe295559f" containerName="maas-api" containerID="cri-o://3880e47f9d102d7a8da2f4915299db7383fca0672ac0b60af29809eaffc8cbe7" gracePeriod=30
Jul 14 15:59:44.115308 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.115234    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:59:44.186317 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.186285    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224b1ae9-152e-4e2e-ade1-4aafe295559f-maas-api-tls\") pod \"224b1ae9-152e-4e2e-ade1-4aafe295559f\" (UID: \"224b1ae9-152e-4e2e-ade1-4aafe295559f\") "
Jul 14 15:59:44.186489 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.186361    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224b1ae9-152e-4e2e-ade1-4aafe295559f-openshift-service-ca\") pod \"224b1ae9-152e-4e2e-ade1-4aafe295559f\" (UID: \"224b1ae9-152e-4e2e-ade1-4aafe295559f\") "
Jul 14 15:59:44.186489 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.186389    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbbnw\" (UniqueName: \"kubernetes.io/projected/224b1ae9-152e-4e2e-ade1-4aafe295559f-kube-api-access-rbbnw\") pod \"224b1ae9-152e-4e2e-ade1-4aafe295559f\" (UID: \"224b1ae9-152e-4e2e-ade1-4aafe295559f\") "
Jul 14 15:59:44.186813 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.186783    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224b1ae9-152e-4e2e-ade1-4aafe295559f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "224b1ae9-152e-4e2e-ade1-4aafe295559f" (UID: "224b1ae9-152e-4e2e-ade1-4aafe295559f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:59:44.188612 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.188585    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224b1ae9-152e-4e2e-ade1-4aafe295559f-kube-api-access-rbbnw" (OuterVolumeSpecName: "kube-api-access-rbbnw") pod "224b1ae9-152e-4e2e-ade1-4aafe295559f" (UID: "224b1ae9-152e-4e2e-ade1-4aafe295559f"). InnerVolumeSpecName "kube-api-access-rbbnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:59:44.188684 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.188631    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224b1ae9-152e-4e2e-ade1-4aafe295559f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "224b1ae9-152e-4e2e-ade1-4aafe295559f" (UID: "224b1ae9-152e-4e2e-ade1-4aafe295559f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:59:44.287682 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.287647    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224b1ae9-152e-4e2e-ade1-4aafe295559f-maas-api-tls\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:44.287682 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.287681    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224b1ae9-152e-4e2e-ade1-4aafe295559f-openshift-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:44.287928 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.287697    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbbnw\" (UniqueName: \"kubernetes.io/projected/224b1ae9-152e-4e2e-ade1-4aafe295559f-kube-api-access-rbbnw\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 15:59:44.397146 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.397109    2563 generic.go:356] "Generic (PLEG): container finished" podID="224b1ae9-152e-4e2e-ade1-4aafe295559f" containerID="3880e47f9d102d7a8da2f4915299db7383fca0672ac0b60af29809eaffc8cbe7" exitCode=0
Jul 14 15:59:44.397308 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.397157    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"
Jul 14 15:59:44.397308 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.397163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5" event={"ID":"224b1ae9-152e-4e2e-ade1-4aafe295559f","Type":"ContainerDied","Data":"3880e47f9d102d7a8da2f4915299db7383fca0672ac0b60af29809eaffc8cbe7"}
Jul 14 15:59:44.397308 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.397221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5" event={"ID":"224b1ae9-152e-4e2e-ade1-4aafe295559f","Type":"ContainerDied","Data":"a5d6b3cae72abe4636e833ba2394ef07d4038a50c8c62cf930ca52e9687989dd"}
Jul 14 15:59:44.397308 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.397249    2563 scope.go:117] "RemoveContainer" containerID="3880e47f9d102d7a8da2f4915299db7383fca0672ac0b60af29809eaffc8cbe7"
Jul 14 15:59:44.411605 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.411580    2563 scope.go:117] "RemoveContainer" containerID="3880e47f9d102d7a8da2f4915299db7383fca0672ac0b60af29809eaffc8cbe7"
Jul 14 15:59:44.411898 ip-10-0-136-210 kubenswrapper[2563]: E0714 15:59:44.411876    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3880e47f9d102d7a8da2f4915299db7383fca0672ac0b60af29809eaffc8cbe7\": container with ID starting with 3880e47f9d102d7a8da2f4915299db7383fca0672ac0b60af29809eaffc8cbe7 not found: ID does not exist" containerID="3880e47f9d102d7a8da2f4915299db7383fca0672ac0b60af29809eaffc8cbe7"
Jul 14 15:59:44.411969 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.411906    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3880e47f9d102d7a8da2f4915299db7383fca0672ac0b60af29809eaffc8cbe7"} err="failed to get container status \"3880e47f9d102d7a8da2f4915299db7383fca0672ac0b60af29809eaffc8cbe7\": rpc error: code = NotFound desc = could not find container \"3880e47f9d102d7a8da2f4915299db7383fca0672ac0b60af29809eaffc8cbe7\": container with ID starting with 3880e47f9d102d7a8da2f4915299db7383fca0672ac0b60af29809eaffc8cbe7 not found: ID does not exist"
Jul 14 15:59:44.424424 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.424397    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"]
Jul 14 15:59:44.426498 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:44.426477    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-81addd-797df5db96-6tzb5"]
Jul 14 15:59:46.216851 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:46.216817    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="224b1ae9-152e-4e2e-ade1-4aafe295559f" path="/var/lib/kubelet/pods/224b1ae9-152e-4e2e-ade1-4aafe295559f/volumes"
Jul 14 15:59:59.204245 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:59.204213    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"]
Jul 14 15:59:59.205163 ip-10-0-136-210 kubenswrapper[2563]: I0714 15:59:59.204417    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd" podUID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" containerName="istio-proxy" containerID="cri-o://d2556619aeb6d00b69135459334fe3ef772eb1ace84d41a88bb2d6d27bfc146c" gracePeriod=30
Jul 14 16:00:00.148634 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.148601    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734080-2xqkr"]
Jul 14 16:00:00.149269 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.149249    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" containerName="istio-proxy"
Jul 14 16:00:00.149327 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.149273    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" containerName="istio-proxy"
Jul 14 16:00:00.149327 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.149310    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d338fbc2-2c01-48dc-8497-8c3e9284faa4" containerName="maas-api"
Jul 14 16:00:00.149327 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.149319    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d338fbc2-2c01-48dc-8497-8c3e9284faa4" containerName="maas-api"
Jul 14 16:00:00.149430 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.149342    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="224b1ae9-152e-4e2e-ade1-4aafe295559f" containerName="maas-api"
Jul 14 16:00:00.149430 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.149352    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="224b1ae9-152e-4e2e-ade1-4aafe295559f" containerName="maas-api"
Jul 14 16:00:00.149430 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.149362    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11074954-387e-4b6d-bb73-726d19215daf" containerName="revoke-keys"
Jul 14 16:00:00.149430 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.149371    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="11074954-387e-4b6d-bb73-726d19215daf" containerName="revoke-keys"
Jul 14 16:00:00.149594 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.149461    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" containerName="istio-proxy"
Jul 14 16:00:00.149594 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.149472    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="11074954-387e-4b6d-bb73-726d19215daf" containerName="revoke-keys"
Jul 14 16:00:00.149594 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.149487    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d338fbc2-2c01-48dc-8497-8c3e9284faa4" containerName="maas-api"
Jul 14 16:00:00.149594 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.149521    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="224b1ae9-152e-4e2e-ade1-4aafe295559f" containerName="maas-api"
Jul 14 16:00:00.153304 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.153282    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"]
Jul 14 16:00:00.153406 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.153392    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734080-2xqkr"
Jul 14 16:00:00.153911 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.153890    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" containerName="istio-proxy"
Jul 14 16:00:00.153911 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.153912    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" containerName="istio-proxy"
Jul 14 16:00:00.154053 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.154041    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5785718-acd7-43b6-a3f5-49f3a5bc04c1" containerName="istio-proxy"
Jul 14 16:00:00.156696 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.156595    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4dcsv\""
Jul 14 16:00:00.161921 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.161870    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734080-2xqkr"]
Jul 14 16:00:00.162024 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.161974    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.164917 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.164872    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734080-2xqkr"
Jul 14 16:00:00.165102 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.165089    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-768d39-openshift-default-dockercfg-v77lk\""
Jul 14 16:00:00.169912 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.169894    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"]
Jul 14 16:00:00.314685 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.314541    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734080-2xqkr"]
Jul 14 16:00:00.317112 ip-10-0-136-210 kubenswrapper[2563]: W0714 16:00:00.317085    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39978635_b618_408c_84d1_8dc2107531e2.slice/crio-7c7d71fd308e93422f05cc76f08a44abf288b37c5f378abd3c6b61c073f7969c WatchSource:0}: Error finding container 7c7d71fd308e93422f05cc76f08a44abf288b37c5f378abd3c6b61c073f7969c: Status 404 returned error can't find the container with id 7c7d71fd308e93422f05cc76f08a44abf288b37c5f378abd3c6b61c073f7969c
Jul 14 16:00:00.345909 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.345883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-workload-certs\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.346023 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.345932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-data\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.346023 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.345959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-podinfo\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.346023 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.345996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-credential-socket\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.346187 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.346024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-envoy\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.346187 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.346081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-token\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.346187 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.346105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vblhc\" (UniqueName: \"kubernetes.io/projected/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-kube-api-access-vblhc\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.346187 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.346148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-workload-socket\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.346187 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.346173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istiod-ca-cert\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.447572 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.447476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-token\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.447572 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.447545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vblhc\" (UniqueName: \"kubernetes.io/projected/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-kube-api-access-vblhc\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.447765 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.447581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-workload-socket\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.447765 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.447611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istiod-ca-cert\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.447765 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.447647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-workload-certs\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.447765 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.447682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-data\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.447765 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.447722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-podinfo\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.447765 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.447757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-credential-socket\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.448050 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.447798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-envoy\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.448050 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.448010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-workload-socket\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.448147 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.448091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-data\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.448213 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.448144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-credential-socket\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.448213 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.448183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-workload-certs\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.448326 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.448312    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istiod-ca-cert\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.450222 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.450199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-envoy\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.450389 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.450369    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-podinfo\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.464672 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.464649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-token\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.464788 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.464770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vblhc\" (UniqueName: \"kubernetes.io/projected/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-kube-api-access-vblhc\") pod \"e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") " pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.476692 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.476677    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:00.492660 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.492632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734080-2xqkr" event={"ID":"39978635-b618-408c-84d1-8dc2107531e2","Type":"ContainerStarted","Data":"7c7d71fd308e93422f05cc76f08a44abf288b37c5f378abd3c6b61c073f7969c"}
Jul 14 16:00:00.623482 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.623447    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"]
Jul 14 16:00:00.638302 ip-10-0-136-210 kubenswrapper[2563]: W0714 16:00:00.638271    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf372d1f4_7ba3_4bdf_82f2_dff21408a95b.slice/crio-bbb98a8a226b70e5970c253bb9c4ffb15cfb64ca4e020ae433264ac844f662df WatchSource:0}: Error finding container bbb98a8a226b70e5970c253bb9c4ffb15cfb64ca4e020ae433264ac844f662df: Status 404 returned error can't find the container with id bbb98a8a226b70e5970c253bb9c4ffb15cfb64ca4e020ae433264ac844f662df
Jul 14 16:00:00.640699 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.640667    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:00:00.640795 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.640735    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:00:00.640795 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:00.640760    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:00:01.500278 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:01.500236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs" event={"ID":"f372d1f4-7ba3-4bdf-82f2-dff21408a95b","Type":"ContainerStarted","Data":"aa97f6ec2516db61073923a60a53e6ab16d5d0e63930072260060ff29aa25d27"}
Jul 14 16:00:01.500717 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:01.500334    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs" event={"ID":"f372d1f4-7ba3-4bdf-82f2-dff21408a95b","Type":"ContainerStarted","Data":"bbb98a8a226b70e5970c253bb9c4ffb15cfb64ca4e020ae433264ac844f662df"}
Jul 14 16:00:01.501736 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:01.501710    2563 generic.go:356] "Generic (PLEG): container finished" podID="39978635-b618-408c-84d1-8dc2107531e2" containerID="a0a347f650c46f30ad63b38d6aa800d769c084eeed9afef9645df13c1b747a71" exitCode=0
Jul 14 16:00:01.501860 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:01.501799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734080-2xqkr" event={"ID":"39978635-b618-408c-84d1-8dc2107531e2","Type":"ContainerDied","Data":"a0a347f650c46f30ad63b38d6aa800d769c084eeed9afef9645df13c1b747a71"}
Jul 14 16:00:01.525037 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:01.524990    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs" podStartSLOduration=1.524975934 podStartE2EDuration="1.524975934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:00:01.52360461 +0000 UTC m=+2325.864870547" watchObservedRunningTime="2026-07-14 16:00:01.524975934 +0000 UTC m=+2325.866241841"
Jul 14 16:00:02.477418 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:02.477386    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:02.481791 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:02.481769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:02.507406 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:02.507378    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:02.508221 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:02.508202    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:02.650103 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:02.650082    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734080-2xqkr"
Jul 14 16:00:03.520021 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.519986    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734080-2xqkr"
Jul 14 16:00:03.520021 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.520002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734080-2xqkr" event={"ID":"39978635-b618-408c-84d1-8dc2107531e2","Type":"ContainerDied","Data":"7c7d71fd308e93422f05cc76f08a44abf288b37c5f378abd3c6b61c073f7969c"}
Jul 14 16:00:03.520479 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.520087    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c7d71fd308e93422f05cc76f08a44abf288b37c5f378abd3c6b61c073f7969c"
Jul 14 16:00:03.576573 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.576544    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"]
Jul 14 16:00:03.576976 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.576962    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39978635-b618-408c-84d1-8dc2107531e2" containerName="cleanup"
Jul 14 16:00:03.577017 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.576977    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="39978635-b618-408c-84d1-8dc2107531e2" containerName="cleanup"
Jul 14 16:00:03.577068 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.577059    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="39978635-b618-408c-84d1-8dc2107531e2" containerName="cleanup"
Jul 14 16:00:03.580484 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.580463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:03.583459 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.583433    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-768d39-serving-cert\""
Jul 14 16:00:03.583696 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.583637    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npnj7\""
Jul 14 16:00:03.583696 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.583659    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 16:00:03.583850 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.583726    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 16:00:03.591763 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.591741    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"]
Jul 14 16:00:03.682821 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.682786    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f9390b-2b08-4827-b662-b2e3c3f166dc-maas-api-tls\") pod \"maas-api-e2e-api-a-768d39-749bbffbd9-g96rp\" (UID: \"50f9390b-2b08-4827-b662-b2e3c3f166dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:03.683021 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.682902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5zjf\" (UniqueName: \"kubernetes.io/projected/50f9390b-2b08-4827-b662-b2e3c3f166dc-kube-api-access-k5zjf\") pod \"maas-api-e2e-api-a-768d39-749bbffbd9-g96rp\" (UID: \"50f9390b-2b08-4827-b662-b2e3c3f166dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:03.683021 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.682957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f9390b-2b08-4827-b662-b2e3c3f166dc-openshift-service-ca\") pod \"maas-api-e2e-api-a-768d39-749bbffbd9-g96rp\" (UID: \"50f9390b-2b08-4827-b662-b2e3c3f166dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:03.784311 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.784218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f9390b-2b08-4827-b662-b2e3c3f166dc-maas-api-tls\") pod \"maas-api-e2e-api-a-768d39-749bbffbd9-g96rp\" (UID: \"50f9390b-2b08-4827-b662-b2e3c3f166dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:03.784311 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.784280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5zjf\" (UniqueName: \"kubernetes.io/projected/50f9390b-2b08-4827-b662-b2e3c3f166dc-kube-api-access-k5zjf\") pod \"maas-api-e2e-api-a-768d39-749bbffbd9-g96rp\" (UID: \"50f9390b-2b08-4827-b662-b2e3c3f166dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:03.784311 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.784309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f9390b-2b08-4827-b662-b2e3c3f166dc-openshift-service-ca\") pod \"maas-api-e2e-api-a-768d39-749bbffbd9-g96rp\" (UID: \"50f9390b-2b08-4827-b662-b2e3c3f166dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:03.784591 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:00:03.784363    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-serving-cert: secret "maas-api-e2e-api-a-768d39-serving-cert" not found
Jul 14 16:00:03.784591 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:00:03.784425    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50f9390b-2b08-4827-b662-b2e3c3f166dc-maas-api-tls podName:50f9390b-2b08-4827-b662-b2e3c3f166dc nodeName:}" failed. No retries permitted until 2026-07-14 16:00:04.284409145 +0000 UTC m=+2328.625675030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50f9390b-2b08-4827-b662-b2e3c3f166dc-maas-api-tls") pod "maas-api-e2e-api-a-768d39-749bbffbd9-g96rp" (UID: "50f9390b-2b08-4827-b662-b2e3c3f166dc") : secret "maas-api-e2e-api-a-768d39-serving-cert" not found
Jul 14 16:00:03.784952 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.784935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f9390b-2b08-4827-b662-b2e3c3f166dc-openshift-service-ca\") pod \"maas-api-e2e-api-a-768d39-749bbffbd9-g96rp\" (UID: \"50f9390b-2b08-4827-b662-b2e3c3f166dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:03.795590 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:03.795568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5zjf\" (UniqueName: \"kubernetes.io/projected/50f9390b-2b08-4827-b662-b2e3c3f166dc-kube-api-access-k5zjf\") pod \"maas-api-e2e-api-a-768d39-749bbffbd9-g96rp\" (UID: \"50f9390b-2b08-4827-b662-b2e3c3f166dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:04.289536 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.289478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f9390b-2b08-4827-b662-b2e3c3f166dc-maas-api-tls\") pod \"maas-api-e2e-api-a-768d39-749bbffbd9-g96rp\" (UID: \"50f9390b-2b08-4827-b662-b2e3c3f166dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:04.291968 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.291945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f9390b-2b08-4827-b662-b2e3c3f166dc-maas-api-tls\") pod \"maas-api-e2e-api-a-768d39-749bbffbd9-g96rp\" (UID: \"50f9390b-2b08-4827-b662-b2e3c3f166dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:04.465955 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.465883    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 16:00:04.492622 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.492593    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:04.529014 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.528977    2563 generic.go:356] "Generic (PLEG): container finished" podID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" containerID="d2556619aeb6d00b69135459334fe3ef772eb1ace84d41a88bb2d6d27bfc146c" exitCode=0
Jul 14 16:00:04.529414 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.529023    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"
Jul 14 16:00:04.529414 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.529058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd" event={"ID":"11bdbff9-93fa-4a57-9f03-e1f0bb01c792","Type":"ContainerDied","Data":"d2556619aeb6d00b69135459334fe3ef772eb1ace84d41a88bb2d6d27bfc146c"}
Jul 14 16:00:04.529414 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.529112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd" event={"ID":"11bdbff9-93fa-4a57-9f03-e1f0bb01c792","Type":"ContainerDied","Data":"e37df31ee83cf9471989c296c1df260a695e6a784341af27464349be7509d5ca"}
Jul 14 16:00:04.529414 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.529135    2563 scope.go:117] "RemoveContainer" containerID="d2556619aeb6d00b69135459334fe3ef772eb1ace84d41a88bb2d6d27bfc146c"
Jul 14 16:00:04.539217 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.539172    2563 scope.go:117] "RemoveContainer" containerID="31ca3c5cbfca599a2a5f7aa99b00a795e95c43cf41bf37275d9553d9b2da25bd"
Jul 14 16:00:04.549303 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.549211    2563 scope.go:117] "RemoveContainer" containerID="d2556619aeb6d00b69135459334fe3ef772eb1ace84d41a88bb2d6d27bfc146c"
Jul 14 16:00:04.549693 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:00:04.549578    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2556619aeb6d00b69135459334fe3ef772eb1ace84d41a88bb2d6d27bfc146c\": container with ID starting with d2556619aeb6d00b69135459334fe3ef772eb1ace84d41a88bb2d6d27bfc146c not found: ID does not exist" containerID="d2556619aeb6d00b69135459334fe3ef772eb1ace84d41a88bb2d6d27bfc146c"
Jul 14 16:00:04.549693 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.549618    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2556619aeb6d00b69135459334fe3ef772eb1ace84d41a88bb2d6d27bfc146c"} err="failed to get container status \"d2556619aeb6d00b69135459334fe3ef772eb1ace84d41a88bb2d6d27bfc146c\": rpc error: code = NotFound desc = could not find container \"d2556619aeb6d00b69135459334fe3ef772eb1ace84d41a88bb2d6d27bfc146c\": container with ID starting with d2556619aeb6d00b69135459334fe3ef772eb1ace84d41a88bb2d6d27bfc146c not found: ID does not exist"
Jul 14 16:00:04.549693 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.549639    2563 scope.go:117] "RemoveContainer" containerID="31ca3c5cbfca599a2a5f7aa99b00a795e95c43cf41bf37275d9553d9b2da25bd"
Jul 14 16:00:04.549951 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:00:04.549928    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ca3c5cbfca599a2a5f7aa99b00a795e95c43cf41bf37275d9553d9b2da25bd\": container with ID starting with 31ca3c5cbfca599a2a5f7aa99b00a795e95c43cf41bf37275d9553d9b2da25bd not found: ID does not exist" containerID="31ca3c5cbfca599a2a5f7aa99b00a795e95c43cf41bf37275d9553d9b2da25bd"
Jul 14 16:00:04.550066 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.549958    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ca3c5cbfca599a2a5f7aa99b00a795e95c43cf41bf37275d9553d9b2da25bd"} err="failed to get container status \"31ca3c5cbfca599a2a5f7aa99b00a795e95c43cf41bf37275d9553d9b2da25bd\": rpc error: code = NotFound desc = could not find container \"31ca3c5cbfca599a2a5f7aa99b00a795e95c43cf41bf37275d9553d9b2da25bd\": container with ID starting with 31ca3c5cbfca599a2a5f7aa99b00a795e95c43cf41bf37275d9553d9b2da25bd not found: ID does not exist"
Jul 14 16:00:04.591975 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.591946    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt5c9\" (UniqueName: \"kubernetes.io/projected/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-kube-api-access-bt5c9\") pod \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") "
Jul 14 16:00:04.592116 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.592004    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-token\") pod \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") "
Jul 14 16:00:04.592116 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.592032    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-credential-socket\") pod \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") "
Jul 14 16:00:04.592116 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.592078    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-podinfo\") pod \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") "
Jul 14 16:00:04.592116 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.592111    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-workload-certs\") pod \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") "
Jul 14 16:00:04.594579 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.592133    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-workload-socket\") pod \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") "
Jul 14 16:00:04.594579 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.592194    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-data\") pod \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") "
Jul 14 16:00:04.594579 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.592216    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istiod-ca-cert\") pod \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") "
Jul 14 16:00:04.594579 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.592268    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-envoy\") pod \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\" (UID: \"11bdbff9-93fa-4a57-9f03-e1f0bb01c792\") "
Jul 14 16:00:04.594579 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.592577    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11bdbff9-93fa-4a57-9f03-e1f0bb01c792" (UID: "11bdbff9-93fa-4a57-9f03-e1f0bb01c792"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:00:04.594579 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.592686    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-workload-certs\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:04.594579 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.592707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11bdbff9-93fa-4a57-9f03-e1f0bb01c792" (UID: "11bdbff9-93fa-4a57-9f03-e1f0bb01c792"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:00:04.594579 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.593418    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-data" (OuterVolumeSpecName: "istio-data") pod "11bdbff9-93fa-4a57-9f03-e1f0bb01c792" (UID: "11bdbff9-93fa-4a57-9f03-e1f0bb01c792"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:00:04.594579 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.593817    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11bdbff9-93fa-4a57-9f03-e1f0bb01c792" (UID: "11bdbff9-93fa-4a57-9f03-e1f0bb01c792"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:00:04.594579 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.593892    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11bdbff9-93fa-4a57-9f03-e1f0bb01c792" (UID: "11bdbff9-93fa-4a57-9f03-e1f0bb01c792"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 16:00:04.595648 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.595610    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11bdbff9-93fa-4a57-9f03-e1f0bb01c792" (UID: "11bdbff9-93fa-4a57-9f03-e1f0bb01c792"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 16:00:04.595762 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.595687    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11bdbff9-93fa-4a57-9f03-e1f0bb01c792" (UID: "11bdbff9-93fa-4a57-9f03-e1f0bb01c792"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:00:04.595762 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.595713    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-kube-api-access-bt5c9" (OuterVolumeSpecName: "kube-api-access-bt5c9") pod "11bdbff9-93fa-4a57-9f03-e1f0bb01c792" (UID: "11bdbff9-93fa-4a57-9f03-e1f0bb01c792"). InnerVolumeSpecName "kube-api-access-bt5c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 16:00:04.597156 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.597121    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-token" (OuterVolumeSpecName: "istio-token") pod "11bdbff9-93fa-4a57-9f03-e1f0bb01c792" (UID: "11bdbff9-93fa-4a57-9f03-e1f0bb01c792"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 16:00:04.633131 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.633107    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"]
Jul 14 16:00:04.634392 ip-10-0-136-210 kubenswrapper[2563]: W0714 16:00:04.634368    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f9390b_2b08_4827_b662_b2e3c3f166dc.slice/crio-d61fea3319bf7dc34bd6db346e8c94aaaa11f316b8319523d4ef0ff8e96af0f3 WatchSource:0}: Error finding container d61fea3319bf7dc34bd6db346e8c94aaaa11f316b8319523d4ef0ff8e96af0f3: Status 404 returned error can't find the container with id d61fea3319bf7dc34bd6db346e8c94aaaa11f316b8319523d4ef0ff8e96af0f3
Jul 14 16:00:04.694028 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.693996    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-envoy\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:04.694028 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.694026    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt5c9\" (UniqueName: \"kubernetes.io/projected/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-kube-api-access-bt5c9\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:04.694274 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.694040    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-token\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:04.694274 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.694055    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-credential-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:04.694274 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.694068    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-podinfo\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:04.694274 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.694083    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-workload-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:04.694274 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.694096    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istio-data\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:04.694274 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.694109    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11bdbff9-93fa-4a57-9f03-e1f0bb01c792-istiod-ca-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:04.864700 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.864670    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"]
Jul 14 16:00:04.871257 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:04.871219    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-81addd-openshift-default-84d58bf587-ntpmd"]
Jul 14 16:00:05.536378 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:05.536348    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp" event={"ID":"50f9390b-2b08-4827-b662-b2e3c3f166dc","Type":"ContainerStarted","Data":"d61fea3319bf7dc34bd6db346e8c94aaaa11f316b8319523d4ef0ff8e96af0f3"}
Jul 14 16:00:06.217994 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:06.217962    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" path="/var/lib/kubelet/pods/11bdbff9-93fa-4a57-9f03-e1f0bb01c792/volumes"
Jul 14 16:00:06.543711 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:06.543615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp" event={"ID":"50f9390b-2b08-4827-b662-b2e3c3f166dc","Type":"ContainerStarted","Data":"106cfb3db755e94d47752c09b460795b806441895853e0fd4f2396959ec97fc3"}
Jul 14 16:00:06.544124 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:06.543740    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:06.563059 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:06.562970    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp" podStartSLOduration=1.9474559089999999 podStartE2EDuration="3.562955396s" totalImagesPullingTime="1.615499487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:00:06.561412869 +0000 UTC m=+2330.902678806" watchObservedRunningTime="2026-07-14 16:00:06.562955396 +0000 UTC m=+2330.904221303"
Jul 14 16:00:08.832304 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.832274    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"]
Jul 14 16:00:08.832933 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.832912    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" containerName="istio-proxy"
Jul 14 16:00:08.833021 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.832936    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" containerName="istio-proxy"
Jul 14 16:00:08.833080 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.833072    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" containerName="istio-proxy"
Jul 14 16:00:08.833129 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.833084    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" containerName="istio-proxy"
Jul 14 16:00:08.836577 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.836558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:08.839197 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.839178    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0e29de-openshift-default-dockercfg-qxqvl\""
Jul 14 16:00:08.848734 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.848709    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"]
Jul 14 16:00:08.933404 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.933372    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c206673-377b-4f43-bfb8-1009c989ba66-istio-token\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:08.933621 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.933431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c206673-377b-4f43-bfb8-1009c989ba66-istiod-ca-cert\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:08.933621 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.933552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-workload-certs\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:08.933621 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.933597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htqfk\" (UniqueName: \"kubernetes.io/projected/0c206673-377b-4f43-bfb8-1009c989ba66-kube-api-access-htqfk\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:08.933778 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.933633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c206673-377b-4f43-bfb8-1009c989ba66-istio-podinfo\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:08.933778 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.933668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-istio-data\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:08.933778 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.933706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-workload-socket\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:08.933778 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.933743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-istio-envoy\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:08.933778 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:08.933764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-credential-socket\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.034915 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.034881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-workload-socket\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.035100 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.034935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-istio-envoy\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.035100 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.034958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-credential-socket\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.035100 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.035003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c206673-377b-4f43-bfb8-1009c989ba66-istio-token\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.035100 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.035031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c206673-377b-4f43-bfb8-1009c989ba66-istiod-ca-cert\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.035100 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.035056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-workload-certs\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.035100 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.035082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htqfk\" (UniqueName: \"kubernetes.io/projected/0c206673-377b-4f43-bfb8-1009c989ba66-kube-api-access-htqfk\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.035433 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.035184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c206673-377b-4f43-bfb8-1009c989ba66-istio-podinfo\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.035433 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.035240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-istio-data\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.035433 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.035353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-workload-socket\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.035433 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.035397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-credential-socket\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.035664 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.035632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-istio-data\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.035720 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.035677    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-workload-certs\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.035936 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.035814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c206673-377b-4f43-bfb8-1009c989ba66-istiod-ca-cert\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.037411 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.037382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-istio-envoy\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.037680 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.037662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c206673-377b-4f43-bfb8-1009c989ba66-istio-podinfo\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.056196 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.056167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c206673-377b-4f43-bfb8-1009c989ba66-istio-token\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.056296 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.056260    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htqfk\" (UniqueName: \"kubernetes.io/projected/0c206673-377b-4f43-bfb8-1009c989ba66-kube-api-access-htqfk\") pod \"e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") " pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.149401 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.149375    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:09.298269 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.298241    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"]
Jul 14 16:00:09.300066 ip-10-0-136-210 kubenswrapper[2563]: W0714 16:00:09.300035    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c206673_377b_4f43_bfb8_1009c989ba66.slice/crio-c1bd05c5474e9dad5d3f832c32d82a73c45d95b9ae9f622df34a74c00b20b0b3 WatchSource:0}: Error finding container c1bd05c5474e9dad5d3f832c32d82a73c45d95b9ae9f622df34a74c00b20b0b3: Status 404 returned error can't find the container with id c1bd05c5474e9dad5d3f832c32d82a73c45d95b9ae9f622df34a74c00b20b0b3
Jul 14 16:00:09.302750 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.302710    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:00:09.302834 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.302780    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:00:09.302834 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.302814    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:00:09.563125 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.563013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n" event={"ID":"0c206673-377b-4f43-bfb8-1009c989ba66","Type":"ContainerStarted","Data":"c9b724a3230411c85a452a21e9c77d9dfa4828e3f012debf7784b19680ef587c"}
Jul 14 16:00:09.563125 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.563087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n" event={"ID":"0c206673-377b-4f43-bfb8-1009c989ba66","Type":"ContainerStarted","Data":"c1bd05c5474e9dad5d3f832c32d82a73c45d95b9ae9f622df34a74c00b20b0b3"}
Jul 14 16:00:09.594289 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:09.594242    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n" podStartSLOduration=1.5942272769999999 podStartE2EDuration="1.594227277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:00:09.592751403 +0000 UTC m=+2333.934017330" watchObservedRunningTime="2026-07-14 16:00:09.594227277 +0000 UTC m=+2333.935493184"
Jul 14 16:00:10.149976 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:10.149940    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:10.154426 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:10.154401    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:10.569271 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:10.569185    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:10.570146 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:10.570126    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:11.723306 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:11.723267    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"]
Jul 14 16:00:11.725536 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:11.723986    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" containerName="istio-proxy"
Jul 14 16:00:11.725536 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:11.724009    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="11bdbff9-93fa-4a57-9f03-e1f0bb01c792" containerName="istio-proxy"
Jul 14 16:00:11.739553 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:11.736326    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"]
Jul 14 16:00:11.739553 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:11.736449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:00:11.746556 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:11.744022    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0e29de-serving-cert\""
Jul 14 16:00:11.765690 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:11.765651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8npt\" (UniqueName: \"kubernetes.io/projected/1d408218-c674-46c4-9d36-b96b01678939-kube-api-access-p8npt\") pod \"maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk\" (UID: \"1d408218-c674-46c4-9d36-b96b01678939\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:00:11.765864 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:11.765769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d408218-c674-46c4-9d36-b96b01678939-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk\" (UID: \"1d408218-c674-46c4-9d36-b96b01678939\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:00:11.765955 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:11.765936    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d408218-c674-46c4-9d36-b96b01678939-maas-api-tls\") pod \"maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk\" (UID: \"1d408218-c674-46c4-9d36-b96b01678939\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:00:11.867315 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:11.867279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d408218-c674-46c4-9d36-b96b01678939-maas-api-tls\") pod \"maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk\" (UID: \"1d408218-c674-46c4-9d36-b96b01678939\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:00:11.867492 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:11.867351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8npt\" (UniqueName: \"kubernetes.io/projected/1d408218-c674-46c4-9d36-b96b01678939-kube-api-access-p8npt\") pod \"maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk\" (UID: \"1d408218-c674-46c4-9d36-b96b01678939\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:00:11.867492 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:11.867383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d408218-c674-46c4-9d36-b96b01678939-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk\" (UID: \"1d408218-c674-46c4-9d36-b96b01678939\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:00:11.867492 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:00:11.867429    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-serving-cert: secret "maas-api-e2e-api-b-0e29de-serving-cert" not found
Jul 14 16:00:11.867704 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:00:11.867495    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d408218-c674-46c4-9d36-b96b01678939-maas-api-tls podName:1d408218-c674-46c4-9d36-b96b01678939 nodeName:}" failed. No retries permitted until 2026-07-14 16:00:12.367478713 +0000 UTC m=+2336.708744598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d408218-c674-46c4-9d36-b96b01678939-maas-api-tls") pod "maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk" (UID: "1d408218-c674-46c4-9d36-b96b01678939") : secret "maas-api-e2e-api-b-0e29de-serving-cert" not found
Jul 14 16:00:11.868015 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:11.867997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d408218-c674-46c4-9d36-b96b01678939-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk\" (UID: \"1d408218-c674-46c4-9d36-b96b01678939\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:00:11.876747 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:11.876717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8npt\" (UniqueName: \"kubernetes.io/projected/1d408218-c674-46c4-9d36-b96b01678939-kube-api-access-p8npt\") pod \"maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk\" (UID: \"1d408218-c674-46c4-9d36-b96b01678939\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:00:12.373705 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:12.373677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d408218-c674-46c4-9d36-b96b01678939-maas-api-tls\") pod \"maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk\" (UID: \"1d408218-c674-46c4-9d36-b96b01678939\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:00:12.376198 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:12.376161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d408218-c674-46c4-9d36-b96b01678939-maas-api-tls\") pod \"maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk\" (UID: \"1d408218-c674-46c4-9d36-b96b01678939\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:00:12.555595 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:12.555566    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:12.664583 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:12.664551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:00:12.877459 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:12.877432    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"]
Jul 14 16:00:12.879312 ip-10-0-136-210 kubenswrapper[2563]: W0714 16:00:12.879286    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d408218_c674_46c4_9d36_b96b01678939.slice/crio-5da862bc8ae73e9f81b860e862b7b2ea8b36b36aeb1a2f3acab2e52ba71ba157 WatchSource:0}: Error finding container 5da862bc8ae73e9f81b860e862b7b2ea8b36b36aeb1a2f3acab2e52ba71ba157: Status 404 returned error can't find the container with id 5da862bc8ae73e9f81b860e862b7b2ea8b36b36aeb1a2f3acab2e52ba71ba157
Jul 14 16:00:13.588241 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:13.588156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk" event={"ID":"1d408218-c674-46c4-9d36-b96b01678939","Type":"ContainerStarted","Data":"95102c49b212d8a39fa3c411b5c113256bca2e69ff7f88cb6fe9e61e5562b073"}
Jul 14 16:00:13.588241 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:13.588216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk" event={"ID":"1d408218-c674-46c4-9d36-b96b01678939","Type":"ContainerStarted","Data":"5da862bc8ae73e9f81b860e862b7b2ea8b36b36aeb1a2f3acab2e52ba71ba157"}
Jul 14 16:00:13.588455 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:13.588248    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:00:13.608096 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:13.608048    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk" podStartSLOduration=2.237070007 podStartE2EDuration="2.608032125s" totalImagesPullingTime="370.962118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:00:13.605769192 +0000 UTC m=+2337.947035138" watchObservedRunningTime="2026-07-14 16:00:13.608032125 +0000 UTC m=+2337.949298032"
Jul 14 16:00:16.606805 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:16.606773    2563 generic.go:356] "Generic (PLEG): container finished" podID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerID="aa97f6ec2516db61073923a60a53e6ab16d5d0e63930072260060ff29aa25d27" exitCode=0
Jul 14 16:00:16.607377 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:16.606811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs" event={"ID":"f372d1f4-7ba3-4bdf-82f2-dff21408a95b","Type":"ContainerDied","Data":"aa97f6ec2516db61073923a60a53e6ab16d5d0e63930072260060ff29aa25d27"}
Jul 14 16:00:16.607377 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:16.607311    2563 scope.go:117] "RemoveContainer" containerID="aa97f6ec2516db61073923a60a53e6ab16d5d0e63930072260060ff29aa25d27"
Jul 14 16:00:16.608481 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:16.608451    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:00:16.608575 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:16.608545    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:00:16.608616 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:16.608585    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:00:17.507360 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:17.507319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:17.615617 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:17.615582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs" event={"ID":"f372d1f4-7ba3-4bdf-82f2-dff21408a95b","Type":"ContainerStarted","Data":"65b49480a3a452514886a4172145b1a3b10da6441ed13cab7c3a0a5016bf4e72"}
Jul 14 16:00:18.477762 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:18.477732    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:18.479060 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:18.479029    2563 patch_prober.go:28] interesting pod/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Jul 14 16:00:18.479208 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:18.479079    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs" podUID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 14 16:00:19.477487 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:19.477449    2563 patch_prober.go:28] interesting pod/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Jul 14 16:00:19.477912 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:19.477533    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs" podUID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 14 16:00:19.599293 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:19.599264    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:00:20.477353 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:20.477315    2563 patch_prober.go:28] interesting pod/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Jul 14 16:00:20.477566 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:20.477366    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs" podUID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 14 16:00:21.477972 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:21.477938    2563 patch_prober.go:28] interesting pod/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Jul 14 16:00:21.478360 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:21.478007    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs" podUID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 14 16:00:22.477984 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:22.477944    2563 patch_prober.go:28] interesting pod/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Jul 14 16:00:22.478470 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:22.478012    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs" podUID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 14 16:00:23.481404 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:23.481370    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:23.481890 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:23.481818    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:23.482582 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:23.482566    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:29.702219 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:29.702189    2563 generic.go:356] "Generic (PLEG): container finished" podID="0c206673-377b-4f43-bfb8-1009c989ba66" containerID="c9b724a3230411c85a452a21e9c77d9dfa4828e3f012debf7784b19680ef587c" exitCode=0
Jul 14 16:00:29.702648 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:29.702265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n" event={"ID":"0c206673-377b-4f43-bfb8-1009c989ba66","Type":"ContainerDied","Data":"c9b724a3230411c85a452a21e9c77d9dfa4828e3f012debf7784b19680ef587c"}
Jul 14 16:00:29.702744 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:29.702728    2563 scope.go:117] "RemoveContainer" containerID="c9b724a3230411c85a452a21e9c77d9dfa4828e3f012debf7784b19680ef587c"
Jul 14 16:00:29.703792 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:29.703764    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:00:29.703918 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:29.703842    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:00:29.703918 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:29.703881    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:00:30.710476 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:30.710438    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n" event={"ID":"0c206673-377b-4f43-bfb8-1009c989ba66","Type":"ContainerStarted","Data":"fd08f2109261a6057c1edfbf1a7fb814dbd29cda51f58037bab6e686c3db12e1"}
Jul 14 16:00:31.149791 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:31.149754    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:31.149969 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:31.149954    2563 patch_prober.go:28] interesting pod/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Jul 14 16:00:31.150015 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:31.149999    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n" podUID="0c206673-377b-4f43-bfb8-1009c989ba66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Jul 14 16:00:32.150238 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:32.150199    2563 patch_prober.go:28] interesting pod/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Jul 14 16:00:32.150665 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:32.150258    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n" podUID="0c206673-377b-4f43-bfb8-1009c989ba66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Jul 14 16:00:33.149758 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:33.149725    2563 patch_prober.go:28] interesting pod/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Jul 14 16:00:33.149952 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:33.149785    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n" podUID="0c206673-377b-4f43-bfb8-1009c989ba66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Jul 14 16:00:34.150481 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:34.150433    2563 patch_prober.go:28] interesting pod/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Jul 14 16:00:34.150905 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:34.150524    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n" podUID="0c206673-377b-4f43-bfb8-1009c989ba66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Jul 14 16:00:35.180598 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.180568    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:35.181081 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.180943    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:35.181691 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.181672    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:00:35.267068 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.267039    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"]
Jul 14 16:00:35.267326 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.267293    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp" podUID="50f9390b-2b08-4827-b662-b2e3c3f166dc" containerName="maas-api" containerID="cri-o://106cfb3db755e94d47752c09b460795b806441895853e0fd4f2396959ec97fc3" gracePeriod=30
Jul 14 16:00:35.540321 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.540294    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:35.612681 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.612653    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5zjf\" (UniqueName: \"kubernetes.io/projected/50f9390b-2b08-4827-b662-b2e3c3f166dc-kube-api-access-k5zjf\") pod \"50f9390b-2b08-4827-b662-b2e3c3f166dc\" (UID: \"50f9390b-2b08-4827-b662-b2e3c3f166dc\") "
Jul 14 16:00:35.612847 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.612695    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f9390b-2b08-4827-b662-b2e3c3f166dc-maas-api-tls\") pod \"50f9390b-2b08-4827-b662-b2e3c3f166dc\" (UID: \"50f9390b-2b08-4827-b662-b2e3c3f166dc\") "
Jul 14 16:00:35.612912 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.612893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f9390b-2b08-4827-b662-b2e3c3f166dc-openshift-service-ca\") pod \"50f9390b-2b08-4827-b662-b2e3c3f166dc\" (UID: \"50f9390b-2b08-4827-b662-b2e3c3f166dc\") "
Jul 14 16:00:35.613224 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.613200    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f9390b-2b08-4827-b662-b2e3c3f166dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "50f9390b-2b08-4827-b662-b2e3c3f166dc" (UID: "50f9390b-2b08-4827-b662-b2e3c3f166dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 16:00:35.614911 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.614883    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f9390b-2b08-4827-b662-b2e3c3f166dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "50f9390b-2b08-4827-b662-b2e3c3f166dc" (UID: "50f9390b-2b08-4827-b662-b2e3c3f166dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 16:00:35.615013 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.614931    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f9390b-2b08-4827-b662-b2e3c3f166dc-kube-api-access-k5zjf" (OuterVolumeSpecName: "kube-api-access-k5zjf") pod "50f9390b-2b08-4827-b662-b2e3c3f166dc" (UID: "50f9390b-2b08-4827-b662-b2e3c3f166dc"). InnerVolumeSpecName "kube-api-access-k5zjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 16:00:35.713812 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.713719    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f9390b-2b08-4827-b662-b2e3c3f166dc-openshift-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:35.713812 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.713750    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5zjf\" (UniqueName: \"kubernetes.io/projected/50f9390b-2b08-4827-b662-b2e3c3f166dc-kube-api-access-k5zjf\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:35.713812 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.713760    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50f9390b-2b08-4827-b662-b2e3c3f166dc-maas-api-tls\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:35.755174 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.755142    2563 generic.go:356] "Generic (PLEG): container finished" podID="50f9390b-2b08-4827-b662-b2e3c3f166dc" containerID="106cfb3db755e94d47752c09b460795b806441895853e0fd4f2396959ec97fc3" exitCode=0
Jul 14 16:00:35.755386 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.755183    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp" event={"ID":"50f9390b-2b08-4827-b662-b2e3c3f166dc","Type":"ContainerDied","Data":"106cfb3db755e94d47752c09b460795b806441895853e0fd4f2396959ec97fc3"}
Jul 14 16:00:35.755386 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.755247    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp" event={"ID":"50f9390b-2b08-4827-b662-b2e3c3f166dc","Type":"ContainerDied","Data":"d61fea3319bf7dc34bd6db346e8c94aaaa11f316b8319523d4ef0ff8e96af0f3"}
Jul 14 16:00:35.755386 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.755195    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"
Jul 14 16:00:35.755386 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.755281    2563 scope.go:117] "RemoveContainer" containerID="106cfb3db755e94d47752c09b460795b806441895853e0fd4f2396959ec97fc3"
Jul 14 16:00:35.764888 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.764871    2563 scope.go:117] "RemoveContainer" containerID="106cfb3db755e94d47752c09b460795b806441895853e0fd4f2396959ec97fc3"
Jul 14 16:00:35.765137 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:00:35.765118    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"106cfb3db755e94d47752c09b460795b806441895853e0fd4f2396959ec97fc3\": container with ID starting with 106cfb3db755e94d47752c09b460795b806441895853e0fd4f2396959ec97fc3 not found: ID does not exist" containerID="106cfb3db755e94d47752c09b460795b806441895853e0fd4f2396959ec97fc3"
Jul 14 16:00:35.765198 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.765148    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"106cfb3db755e94d47752c09b460795b806441895853e0fd4f2396959ec97fc3"} err="failed to get container status \"106cfb3db755e94d47752c09b460795b806441895853e0fd4f2396959ec97fc3\": rpc error: code = NotFound desc = could not find container \"106cfb3db755e94d47752c09b460795b806441895853e0fd4f2396959ec97fc3\": container with ID starting with 106cfb3db755e94d47752c09b460795b806441895853e0fd4f2396959ec97fc3 not found: ID does not exist"
Jul 14 16:00:35.779038 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.779014    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"]
Jul 14 16:00:35.784644 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:35.784626    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-768d39-749bbffbd9-g96rp"]
Jul 14 16:00:36.216944 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:36.216903    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f9390b-2b08-4827-b662-b2e3c3f166dc" path="/var/lib/kubelet/pods/50f9390b-2b08-4827-b662-b2e3c3f166dc/volumes"
Jul 14 16:00:50.243101 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:50.243069    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"]
Jul 14 16:00:50.243614 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:50.243310    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs" podUID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerName="istio-proxy" containerID="cri-o://65b49480a3a452514886a4172145b1a3b10da6441ed13cab7c3a0a5016bf4e72" gracePeriod=30
Jul 14 16:00:53.483547 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:53.483484    2563 patch_prober.go:28] interesting pod/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 16:00:53.483966 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:53.483572    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs" podUID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 16:00:55.501952 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.501923    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:55.507734 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.507682    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istiod-ca-cert\") pod \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") "
Jul 14 16:00:55.507734 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.507717    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-envoy\") pod \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") "
Jul 14 16:00:55.507845 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.507761    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-token\") pod \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") "
Jul 14 16:00:55.507845 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.507804    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-podinfo\") pod \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") "
Jul 14 16:00:55.507845 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.507828    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-data\") pod \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") "
Jul 14 16:00:55.508045 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.507844    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-workload-certs\") pod \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") "
Jul 14 16:00:55.508045 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.507908    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-workload-socket\") pod \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") "
Jul 14 16:00:55.508045 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.507990    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vblhc\" (UniqueName: \"kubernetes.io/projected/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-kube-api-access-vblhc\") pod \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") "
Jul 14 16:00:55.508045 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.508020    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-credential-socket\") pod \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\" (UID: \"f372d1f4-7ba3-4bdf-82f2-dff21408a95b\") "
Jul 14 16:00:55.508263 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.508065    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f372d1f4-7ba3-4bdf-82f2-dff21408a95b" (UID: "f372d1f4-7ba3-4bdf-82f2-dff21408a95b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 16:00:55.508263 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.508170    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f372d1f4-7ba3-4bdf-82f2-dff21408a95b" (UID: "f372d1f4-7ba3-4bdf-82f2-dff21408a95b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:00:55.508395 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.508375    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istiod-ca-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:55.508453 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.508401    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-workload-certs\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:55.508453 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.508411    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f372d1f4-7ba3-4bdf-82f2-dff21408a95b" (UID: "f372d1f4-7ba3-4bdf-82f2-dff21408a95b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:00:55.508750 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.508661    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f372d1f4-7ba3-4bdf-82f2-dff21408a95b" (UID: "f372d1f4-7ba3-4bdf-82f2-dff21408a95b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:00:55.508947 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.508924    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-data" (OuterVolumeSpecName: "istio-data") pod "f372d1f4-7ba3-4bdf-82f2-dff21408a95b" (UID: "f372d1f4-7ba3-4bdf-82f2-dff21408a95b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:00:55.510213 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.510193    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f372d1f4-7ba3-4bdf-82f2-dff21408a95b" (UID: "f372d1f4-7ba3-4bdf-82f2-dff21408a95b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 16:00:55.510335 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.510317    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-token" (OuterVolumeSpecName: "istio-token") pod "f372d1f4-7ba3-4bdf-82f2-dff21408a95b" (UID: "f372d1f4-7ba3-4bdf-82f2-dff21408a95b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 16:00:55.510569 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.510539    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f372d1f4-7ba3-4bdf-82f2-dff21408a95b" (UID: "f372d1f4-7ba3-4bdf-82f2-dff21408a95b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:00:55.510681 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.510624    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-kube-api-access-vblhc" (OuterVolumeSpecName: "kube-api-access-vblhc") pod "f372d1f4-7ba3-4bdf-82f2-dff21408a95b" (UID: "f372d1f4-7ba3-4bdf-82f2-dff21408a95b"). InnerVolumeSpecName "kube-api-access-vblhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 16:00:55.609337 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.609301    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-token\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:55.609337 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.609329    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-podinfo\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:55.609337 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.609338    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-data\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:55.609337 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.609346    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-workload-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:55.609646 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.609355    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vblhc\" (UniqueName: \"kubernetes.io/projected/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-kube-api-access-vblhc\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:55.609646 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.609363    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-credential-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:55.609646 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.609371    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f372d1f4-7ba3-4bdf-82f2-dff21408a95b-istio-envoy\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:00:55.872924 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.872822    2563 generic.go:356] "Generic (PLEG): container finished" podID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerID="65b49480a3a452514886a4172145b1a3b10da6441ed13cab7c3a0a5016bf4e72" exitCode=0
Jul 14 16:00:55.872924 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.872878    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"
Jul 14 16:00:55.872924 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.872903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs" event={"ID":"f372d1f4-7ba3-4bdf-82f2-dff21408a95b","Type":"ContainerDied","Data":"65b49480a3a452514886a4172145b1a3b10da6441ed13cab7c3a0a5016bf4e72"}
Jul 14 16:00:55.873212 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.872969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs" event={"ID":"f372d1f4-7ba3-4bdf-82f2-dff21408a95b","Type":"ContainerDied","Data":"bbb98a8a226b70e5970c253bb9c4ffb15cfb64ca4e020ae433264ac844f662df"}
Jul 14 16:00:55.873212 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.872999    2563 scope.go:117] "RemoveContainer" containerID="65b49480a3a452514886a4172145b1a3b10da6441ed13cab7c3a0a5016bf4e72"
Jul 14 16:00:55.883015 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.882998    2563 scope.go:117] "RemoveContainer" containerID="aa97f6ec2516db61073923a60a53e6ab16d5d0e63930072260060ff29aa25d27"
Jul 14 16:00:55.892143 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.892125    2563 scope.go:117] "RemoveContainer" containerID="65b49480a3a452514886a4172145b1a3b10da6441ed13cab7c3a0a5016bf4e72"
Jul 14 16:00:55.892407 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:00:55.892389    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b49480a3a452514886a4172145b1a3b10da6441ed13cab7c3a0a5016bf4e72\": container with ID starting with 65b49480a3a452514886a4172145b1a3b10da6441ed13cab7c3a0a5016bf4e72 not found: ID does not exist" containerID="65b49480a3a452514886a4172145b1a3b10da6441ed13cab7c3a0a5016bf4e72"
Jul 14 16:00:55.892461 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.892424    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b49480a3a452514886a4172145b1a3b10da6441ed13cab7c3a0a5016bf4e72"} err="failed to get container status \"65b49480a3a452514886a4172145b1a3b10da6441ed13cab7c3a0a5016bf4e72\": rpc error: code = NotFound desc = could not find container \"65b49480a3a452514886a4172145b1a3b10da6441ed13cab7c3a0a5016bf4e72\": container with ID starting with 65b49480a3a452514886a4172145b1a3b10da6441ed13cab7c3a0a5016bf4e72 not found: ID does not exist"
Jul 14 16:00:55.892461 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.892441    2563 scope.go:117] "RemoveContainer" containerID="aa97f6ec2516db61073923a60a53e6ab16d5d0e63930072260060ff29aa25d27"
Jul 14 16:00:55.892690 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:00:55.892662    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa97f6ec2516db61073923a60a53e6ab16d5d0e63930072260060ff29aa25d27\": container with ID starting with aa97f6ec2516db61073923a60a53e6ab16d5d0e63930072260060ff29aa25d27 not found: ID does not exist" containerID="aa97f6ec2516db61073923a60a53e6ab16d5d0e63930072260060ff29aa25d27"
Jul 14 16:00:55.892744 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.892692    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa97f6ec2516db61073923a60a53e6ab16d5d0e63930072260060ff29aa25d27"} err="failed to get container status \"aa97f6ec2516db61073923a60a53e6ab16d5d0e63930072260060ff29aa25d27\": rpc error: code = NotFound desc = could not find container \"aa97f6ec2516db61073923a60a53e6ab16d5d0e63930072260060ff29aa25d27\": container with ID starting with aa97f6ec2516db61073923a60a53e6ab16d5d0e63930072260060ff29aa25d27 not found: ID does not exist"
Jul 14 16:00:55.898275 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.898256    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"]
Jul 14 16:00:55.904654 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:55.904633    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-768d39-openshift-default-6569f7bb8f-58hjs"]
Jul 14 16:00:56.216630 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:00:56.216601    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" path="/var/lib/kubelet/pods/f372d1f4-7ba3-4bdf-82f2-dff21408a95b/volumes"
Jul 14 16:01:01.153778 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.153742    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"]
Jul 14 16:01:01.154585 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.154552    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk" podUID="1d408218-c674-46c4-9d36-b96b01678939" containerName="maas-api" containerID="cri-o://95102c49b212d8a39fa3c411b5c113256bca2e69ff7f88cb6fe9e61e5562b073" gracePeriod=30
Jul 14 16:01:01.419844 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.419764    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:01:01.464365 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.464333    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8npt\" (UniqueName: \"kubernetes.io/projected/1d408218-c674-46c4-9d36-b96b01678939-kube-api-access-p8npt\") pod \"1d408218-c674-46c4-9d36-b96b01678939\" (UID: \"1d408218-c674-46c4-9d36-b96b01678939\") "
Jul 14 16:01:01.464583 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.464451    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d408218-c674-46c4-9d36-b96b01678939-openshift-service-ca\") pod \"1d408218-c674-46c4-9d36-b96b01678939\" (UID: \"1d408218-c674-46c4-9d36-b96b01678939\") "
Jul 14 16:01:01.464583 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.464476    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d408218-c674-46c4-9d36-b96b01678939-maas-api-tls\") pod \"1d408218-c674-46c4-9d36-b96b01678939\" (UID: \"1d408218-c674-46c4-9d36-b96b01678939\") "
Jul 14 16:01:01.464844 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.464821    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d408218-c674-46c4-9d36-b96b01678939-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1d408218-c674-46c4-9d36-b96b01678939" (UID: "1d408218-c674-46c4-9d36-b96b01678939"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 16:01:01.466660 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.466623    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d408218-c674-46c4-9d36-b96b01678939-kube-api-access-p8npt" (OuterVolumeSpecName: "kube-api-access-p8npt") pod "1d408218-c674-46c4-9d36-b96b01678939" (UID: "1d408218-c674-46c4-9d36-b96b01678939"). InnerVolumeSpecName "kube-api-access-p8npt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 16:01:01.466771 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.466704    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d408218-c674-46c4-9d36-b96b01678939-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1d408218-c674-46c4-9d36-b96b01678939" (UID: "1d408218-c674-46c4-9d36-b96b01678939"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 16:01:01.565394 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.565360    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d408218-c674-46c4-9d36-b96b01678939-openshift-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:01:01.565394 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.565388    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d408218-c674-46c4-9d36-b96b01678939-maas-api-tls\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:01:01.565394 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.565399    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8npt\" (UniqueName: \"kubernetes.io/projected/1d408218-c674-46c4-9d36-b96b01678939-kube-api-access-p8npt\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:01:01.911256 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.911216    2563 generic.go:356] "Generic (PLEG): container finished" podID="1d408218-c674-46c4-9d36-b96b01678939" containerID="95102c49b212d8a39fa3c411b5c113256bca2e69ff7f88cb6fe9e61e5562b073" exitCode=0
Jul 14 16:01:01.911447 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.911272    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"
Jul 14 16:01:01.911447 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.911301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk" event={"ID":"1d408218-c674-46c4-9d36-b96b01678939","Type":"ContainerDied","Data":"95102c49b212d8a39fa3c411b5c113256bca2e69ff7f88cb6fe9e61e5562b073"}
Jul 14 16:01:01.911447 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.911351    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk" event={"ID":"1d408218-c674-46c4-9d36-b96b01678939","Type":"ContainerDied","Data":"5da862bc8ae73e9f81b860e862b7b2ea8b36b36aeb1a2f3acab2e52ba71ba157"}
Jul 14 16:01:01.911447 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.911373    2563 scope.go:117] "RemoveContainer" containerID="95102c49b212d8a39fa3c411b5c113256bca2e69ff7f88cb6fe9e61e5562b073"
Jul 14 16:01:01.921454 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.921433    2563 scope.go:117] "RemoveContainer" containerID="95102c49b212d8a39fa3c411b5c113256bca2e69ff7f88cb6fe9e61e5562b073"
Jul 14 16:01:01.921806 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:01:01.921780    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95102c49b212d8a39fa3c411b5c113256bca2e69ff7f88cb6fe9e61e5562b073\": container with ID starting with 95102c49b212d8a39fa3c411b5c113256bca2e69ff7f88cb6fe9e61e5562b073 not found: ID does not exist" containerID="95102c49b212d8a39fa3c411b5c113256bca2e69ff7f88cb6fe9e61e5562b073"
Jul 14 16:01:01.921911 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.921813    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95102c49b212d8a39fa3c411b5c113256bca2e69ff7f88cb6fe9e61e5562b073"} err="failed to get container status \"95102c49b212d8a39fa3c411b5c113256bca2e69ff7f88cb6fe9e61e5562b073\": rpc error: code = NotFound desc = could not find container \"95102c49b212d8a39fa3c411b5c113256bca2e69ff7f88cb6fe9e61e5562b073\": container with ID starting with 95102c49b212d8a39fa3c411b5c113256bca2e69ff7f88cb6fe9e61e5562b073 not found: ID does not exist"
Jul 14 16:01:01.936885 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.936858    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"]
Jul 14 16:01:01.942126 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:01.942101    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e29de-7b6c7f758-bxqhk"]
Jul 14 16:01:02.217049 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:02.216953    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d408218-c674-46c4-9d36-b96b01678939" path="/var/lib/kubelet/pods/1d408218-c674-46c4-9d36-b96b01678939/volumes"
Jul 14 16:01:16.286454 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:16.286424    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 16:01:16.292111 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:16.292091    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 16:01:16.427609 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:16.427578    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"]
Jul 14 16:01:16.427897 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:16.427845    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n" podUID="0c206673-377b-4f43-bfb8-1009c989ba66" containerName="istio-proxy" containerID="cri-o://fd08f2109261a6057c1edfbf1a7fb814dbd29cda51f58037bab6e686c3db12e1" gracePeriod=30
Jul 14 16:01:17.365782 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.365746    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"]
Jul 14 16:01:17.367530 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.367232    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerName="istio-proxy"
Jul 14 16:01:17.367530 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.367275    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerName="istio-proxy"
Jul 14 16:01:17.367530 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.367302    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f9390b-2b08-4827-b662-b2e3c3f166dc" containerName="maas-api"
Jul 14 16:01:17.367530 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.367311    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f9390b-2b08-4827-b662-b2e3c3f166dc" containerName="maas-api"
Jul 14 16:01:17.367530 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.367342    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d408218-c674-46c4-9d36-b96b01678939" containerName="maas-api"
Jul 14 16:01:17.367530 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.367351    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d408218-c674-46c4-9d36-b96b01678939" containerName="maas-api"
Jul 14 16:01:17.367530 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.367361    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerName="istio-proxy"
Jul 14 16:01:17.367530 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.367369    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerName="istio-proxy"
Jul 14 16:01:17.368007 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.367623    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f9390b-2b08-4827-b662-b2e3c3f166dc" containerName="maas-api"
Jul 14 16:01:17.368007 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.367645    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerName="istio-proxy"
Jul 14 16:01:17.368007 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.367656    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f372d1f4-7ba3-4bdf-82f2-dff21408a95b" containerName="istio-proxy"
Jul 14 16:01:17.368007 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.367672    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d408218-c674-46c4-9d36-b96b01678939" containerName="maas-api"
Jul 14 16:01:17.375001 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.371911    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.375189 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.375162    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-bea800-openshift-default-dockercfg-kcn5s\""
Jul 14 16:01:17.382884 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.382796    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"]
Jul 14 16:01:17.525645 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.525592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-credential-socket\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.525842 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.525665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-data\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.525842 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.525697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd5f36b-93a6-456c-8d55-c785a4a98869-istiod-ca-cert\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.525842 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.525722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-envoy\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.526002 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.525832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-token\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.526002 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.525891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-workload-socket\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.526002 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.525923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-podinfo\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.526002 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.525956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-workload-certs\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.526002 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.525993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj5ws\" (UniqueName: \"kubernetes.io/projected/afd5f36b-93a6-456c-8d55-c785a4a98869-kube-api-access-wj5ws\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.627646 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.627541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-credential-socket\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.627646 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.627594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-data\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.627646 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.627614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd5f36b-93a6-456c-8d55-c785a4a98869-istiod-ca-cert\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.627646 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.627630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-envoy\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.627974 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.627661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-token\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.627974 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.627688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-workload-socket\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.627974 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.627718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-podinfo\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.627974 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.627930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-credential-socket\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.628175 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.627982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-workload-certs\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.628175 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.628043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj5ws\" (UniqueName: \"kubernetes.io/projected/afd5f36b-93a6-456c-8d55-c785a4a98869-kube-api-access-wj5ws\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.628356 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.628335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd5f36b-93a6-456c-8d55-c785a4a98869-istiod-ca-cert\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.628435 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.628365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-data\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.628435 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.628411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-workload-socket\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.628693 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.628672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-workload-certs\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.630748 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.630724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-envoy\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.630929 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.630912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-podinfo\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.640995 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.640970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-token\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.641257 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.641214    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj5ws\" (UniqueName: \"kubernetes.io/projected/afd5f36b-93a6-456c-8d55-c785a4a98869-kube-api-access-wj5ws\") pod \"e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") " pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.688452 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.688416    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:17.830766 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.830737    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"]
Jul 14 16:01:17.831549 ip-10-0-136-210 kubenswrapper[2563]: W0714 16:01:17.831527    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd5f36b_93a6_456c_8d55_c785a4a98869.slice/crio-30e941eb90ef8d599377c0d63ab80b979d5186c7ef46621f8aaa38014b77d92b WatchSource:0}: Error finding container 30e941eb90ef8d599377c0d63ab80b979d5186c7ef46621f8aaa38014b77d92b: Status 404 returned error can't find the container with id 30e941eb90ef8d599377c0d63ab80b979d5186c7ef46621f8aaa38014b77d92b
Jul 14 16:01:17.833667 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.833635    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:01:17.833734 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.833706    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:01:17.833772 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:17.833735    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:01:18.003159 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:18.003127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8" event={"ID":"afd5f36b-93a6-456c-8d55-c785a4a98869","Type":"ContainerStarted","Data":"89617fa662aabe89616dcad7c869b01733b8657106c944c54f76aed145ffabab"}
Jul 14 16:01:18.003332 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:18.003177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8" event={"ID":"afd5f36b-93a6-456c-8d55-c785a4a98869","Type":"ContainerStarted","Data":"30e941eb90ef8d599377c0d63ab80b979d5186c7ef46621f8aaa38014b77d92b"}
Jul 14 16:01:18.025648 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:18.025601    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8" podStartSLOduration=1.025582038 podStartE2EDuration="1.025582038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:01:18.023370662 +0000 UTC m=+2402.364636593" watchObservedRunningTime="2026-07-14 16:01:18.025582038 +0000 UTC m=+2402.366847986"
Jul 14 16:01:18.689496 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:18.689454    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:18.694448 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:18.694423    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:19.008667 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:19.008579    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:19.009680 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:19.009664    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:20.099301 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.099258    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"]
Jul 14 16:01:20.102368 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.102343    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:01:20.105850 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.105824    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 16:01:20.106014 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.105870    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-bea800-serving-cert\""
Jul 14 16:01:20.106014 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.105822    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 16:01:20.107119 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.107102    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npnj7\""
Jul 14 16:01:20.112642 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.112618    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"]
Jul 14 16:01:20.183162 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.183127    2563 patch_prober.go:28] interesting pod/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 16:01:20.183328 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.183176    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n" podUID="0c206673-377b-4f43-bfb8-1009c989ba66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 16:01:20.254498 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.254464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a50bda-98a9-4058-ab8a-fd1d16df1887-maas-api-tls\") pod \"maas-api-e2e-shared-a-bea800-86f7466bb9-spt88\" (UID: \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:01:20.254701 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.254534    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a50bda-98a9-4058-ab8a-fd1d16df1887-openshift-service-ca\") pod \"maas-api-e2e-shared-a-bea800-86f7466bb9-spt88\" (UID: \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:01:20.254701 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.254578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nhf8\" (UniqueName: \"kubernetes.io/projected/b5a50bda-98a9-4058-ab8a-fd1d16df1887-kube-api-access-8nhf8\") pod \"maas-api-e2e-shared-a-bea800-86f7466bb9-spt88\" (UID: \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:01:20.355432 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.355331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a50bda-98a9-4058-ab8a-fd1d16df1887-maas-api-tls\") pod \"maas-api-e2e-shared-a-bea800-86f7466bb9-spt88\" (UID: \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:01:20.355432 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.355369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a50bda-98a9-4058-ab8a-fd1d16df1887-openshift-service-ca\") pod \"maas-api-e2e-shared-a-bea800-86f7466bb9-spt88\" (UID: \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:01:20.355432 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.355401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nhf8\" (UniqueName: \"kubernetes.io/projected/b5a50bda-98a9-4058-ab8a-fd1d16df1887-kube-api-access-8nhf8\") pod \"maas-api-e2e-shared-a-bea800-86f7466bb9-spt88\" (UID: \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:01:20.356031 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.355999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a50bda-98a9-4058-ab8a-fd1d16df1887-openshift-service-ca\") pod \"maas-api-e2e-shared-a-bea800-86f7466bb9-spt88\" (UID: \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:01:20.358102 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.358083    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a50bda-98a9-4058-ab8a-fd1d16df1887-maas-api-tls\") pod \"maas-api-e2e-shared-a-bea800-86f7466bb9-spt88\" (UID: \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:01:20.365250 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.365221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nhf8\" (UniqueName: \"kubernetes.io/projected/b5a50bda-98a9-4058-ab8a-fd1d16df1887-kube-api-access-8nhf8\") pod \"maas-api-e2e-shared-a-bea800-86f7466bb9-spt88\" (UID: \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:01:20.415486 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.415435    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:01:20.559071 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:20.559044    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"]
Jul 14 16:01:20.561562 ip-10-0-136-210 kubenswrapper[2563]: W0714 16:01:20.561535    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a50bda_98a9_4058_ab8a_fd1d16df1887.slice/crio-8d187d39ec4391619670ecf9ecd294ce6dff24861362123ce9b34c83b769d984 WatchSource:0}: Error finding container 8d187d39ec4391619670ecf9ecd294ce6dff24861362123ce9b34c83b769d984: Status 404 returned error can't find the container with id 8d187d39ec4391619670ecf9ecd294ce6dff24861362123ce9b34c83b769d984
Jul 14 16:01:21.021145 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.021110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88" event={"ID":"b5a50bda-98a9-4058-ab8a-fd1d16df1887","Type":"ContainerStarted","Data":"8d187d39ec4391619670ecf9ecd294ce6dff24861362123ce9b34c83b769d984"}
Jul 14 16:01:21.679871 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.679847    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:01:21.770187 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c206673-377b-4f43-bfb8-1009c989ba66-istio-token\") pod \"0c206673-377b-4f43-bfb8-1009c989ba66\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") "
Jul 14 16:01:21.770351 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770189    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c206673-377b-4f43-bfb8-1009c989ba66-istiod-ca-cert\") pod \"0c206673-377b-4f43-bfb8-1009c989ba66\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") "
Jul 14 16:01:21.770351 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770248    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-workload-certs\") pod \"0c206673-377b-4f43-bfb8-1009c989ba66\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") "
Jul 14 16:01:21.770351 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770287    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-credential-socket\") pod \"0c206673-377b-4f43-bfb8-1009c989ba66\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") "
Jul 14 16:01:21.770351 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770313    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-workload-socket\") pod \"0c206673-377b-4f43-bfb8-1009c989ba66\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") "
Jul 14 16:01:21.770351 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770338    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htqfk\" (UniqueName: \"kubernetes.io/projected/0c206673-377b-4f43-bfb8-1009c989ba66-kube-api-access-htqfk\") pod \"0c206673-377b-4f43-bfb8-1009c989ba66\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") "
Jul 14 16:01:21.770636 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770394    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-istio-data\") pod \"0c206673-377b-4f43-bfb8-1009c989ba66\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") "
Jul 14 16:01:21.770636 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770432    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-istio-envoy\") pod \"0c206673-377b-4f43-bfb8-1009c989ba66\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") "
Jul 14 16:01:21.770636 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770461    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c206673-377b-4f43-bfb8-1009c989ba66-istio-podinfo\") pod \"0c206673-377b-4f43-bfb8-1009c989ba66\" (UID: \"0c206673-377b-4f43-bfb8-1009c989ba66\") "
Jul 14 16:01:21.770797 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770607    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c206673-377b-4f43-bfb8-1009c989ba66" (UID: "0c206673-377b-4f43-bfb8-1009c989ba66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:01:21.770797 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770713    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c206673-377b-4f43-bfb8-1009c989ba66" (UID: "0c206673-377b-4f43-bfb8-1009c989ba66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:01:21.770797 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770739    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c206673-377b-4f43-bfb8-1009c989ba66-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c206673-377b-4f43-bfb8-1009c989ba66" (UID: "0c206673-377b-4f43-bfb8-1009c989ba66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 16:01:21.770797 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770743    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c206673-377b-4f43-bfb8-1009c989ba66" (UID: "0c206673-377b-4f43-bfb8-1009c989ba66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:01:21.771008 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770880    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c206673-377b-4f43-bfb8-1009c989ba66-istiod-ca-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:01:21.771008 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770900    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-workload-certs\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:01:21.771008 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770916    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-credential-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:01:21.771008 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.770929    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-workload-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:01:21.771724 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.771699    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c206673-377b-4f43-bfb8-1009c989ba66" (UID: "0c206673-377b-4f43-bfb8-1009c989ba66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:01:21.773137 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.773104    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c206673-377b-4f43-bfb8-1009c989ba66" (UID: "0c206673-377b-4f43-bfb8-1009c989ba66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:01:21.773253 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.773169    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c206673-377b-4f43-bfb8-1009c989ba66-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c206673-377b-4f43-bfb8-1009c989ba66" (UID: "0c206673-377b-4f43-bfb8-1009c989ba66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 16:01:21.773436 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.773410    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c206673-377b-4f43-bfb8-1009c989ba66-kube-api-access-htqfk" (OuterVolumeSpecName: "kube-api-access-htqfk") pod "0c206673-377b-4f43-bfb8-1009c989ba66" (UID: "0c206673-377b-4f43-bfb8-1009c989ba66"). InnerVolumeSpecName "kube-api-access-htqfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 16:01:21.773689 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.773670    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c206673-377b-4f43-bfb8-1009c989ba66-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c206673-377b-4f43-bfb8-1009c989ba66" (UID: "0c206673-377b-4f43-bfb8-1009c989ba66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 16:01:21.872298 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.872258    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-istio-data\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:01:21.872298 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.872295    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c206673-377b-4f43-bfb8-1009c989ba66-istio-envoy\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:01:21.872298 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.872304    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c206673-377b-4f43-bfb8-1009c989ba66-istio-podinfo\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:01:21.872565 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.872312    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c206673-377b-4f43-bfb8-1009c989ba66-istio-token\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:01:21.872565 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:21.872321    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htqfk\" (UniqueName: \"kubernetes.io/projected/0c206673-377b-4f43-bfb8-1009c989ba66-kube-api-access-htqfk\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:01:22.027864 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.027774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88" event={"ID":"b5a50bda-98a9-4058-ab8a-fd1d16df1887","Type":"ContainerStarted","Data":"3c8dc4539e0efd7223bd5ae8c086ad87cc40868a1aa4e5bdf7c6b93ae7e2696a"}
Jul 14 16:01:22.028033 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.027905    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:01:22.029492 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.029468    2563 generic.go:356] "Generic (PLEG): container finished" podID="0c206673-377b-4f43-bfb8-1009c989ba66" containerID="fd08f2109261a6057c1edfbf1a7fb814dbd29cda51f58037bab6e686c3db12e1" exitCode=0
Jul 14 16:01:22.029613 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.029534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n" event={"ID":"0c206673-377b-4f43-bfb8-1009c989ba66","Type":"ContainerDied","Data":"fd08f2109261a6057c1edfbf1a7fb814dbd29cda51f58037bab6e686c3db12e1"}
Jul 14 16:01:22.029613 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.029555    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"
Jul 14 16:01:22.029613 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.029581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n" event={"ID":"0c206673-377b-4f43-bfb8-1009c989ba66","Type":"ContainerDied","Data":"c1bd05c5474e9dad5d3f832c32d82a73c45d95b9ae9f622df34a74c00b20b0b3"}
Jul 14 16:01:22.029613 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.029603    2563 scope.go:117] "RemoveContainer" containerID="fd08f2109261a6057c1edfbf1a7fb814dbd29cda51f58037bab6e686c3db12e1"
Jul 14 16:01:22.039942 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.039921    2563 scope.go:117] "RemoveContainer" containerID="c9b724a3230411c85a452a21e9c77d9dfa4828e3f012debf7784b19680ef587c"
Jul 14 16:01:22.048934 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.048879    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88" podStartSLOduration=1.618888047 podStartE2EDuration="2.04886065s" totalImagesPullingTime="429.972603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:01:22.045443666 +0000 UTC m=+2406.386709602" watchObservedRunningTime="2026-07-14 16:01:22.04886065 +0000 UTC m=+2406.390126558"
Jul 14 16:01:22.053002 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.052979    2563 scope.go:117] "RemoveContainer" containerID="fd08f2109261a6057c1edfbf1a7fb814dbd29cda51f58037bab6e686c3db12e1"
Jul 14 16:01:22.053415 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:01:22.053390    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd08f2109261a6057c1edfbf1a7fb814dbd29cda51f58037bab6e686c3db12e1\": container with ID starting with fd08f2109261a6057c1edfbf1a7fb814dbd29cda51f58037bab6e686c3db12e1 not found: ID does not exist" containerID="fd08f2109261a6057c1edfbf1a7fb814dbd29cda51f58037bab6e686c3db12e1"
Jul 14 16:01:22.053488 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.053428    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd08f2109261a6057c1edfbf1a7fb814dbd29cda51f58037bab6e686c3db12e1"} err="failed to get container status \"fd08f2109261a6057c1edfbf1a7fb814dbd29cda51f58037bab6e686c3db12e1\": rpc error: code = NotFound desc = could not find container \"fd08f2109261a6057c1edfbf1a7fb814dbd29cda51f58037bab6e686c3db12e1\": container with ID starting with fd08f2109261a6057c1edfbf1a7fb814dbd29cda51f58037bab6e686c3db12e1 not found: ID does not exist"
Jul 14 16:01:22.053488 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.053445    2563 scope.go:117] "RemoveContainer" containerID="c9b724a3230411c85a452a21e9c77d9dfa4828e3f012debf7784b19680ef587c"
Jul 14 16:01:22.053823 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:01:22.053805    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b724a3230411c85a452a21e9c77d9dfa4828e3f012debf7784b19680ef587c\": container with ID starting with c9b724a3230411c85a452a21e9c77d9dfa4828e3f012debf7784b19680ef587c not found: ID does not exist" containerID="c9b724a3230411c85a452a21e9c77d9dfa4828e3f012debf7784b19680ef587c"
Jul 14 16:01:22.053867 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.053827    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b724a3230411c85a452a21e9c77d9dfa4828e3f012debf7784b19680ef587c"} err="failed to get container status \"c9b724a3230411c85a452a21e9c77d9dfa4828e3f012debf7784b19680ef587c\": rpc error: code = NotFound desc = could not find container \"c9b724a3230411c85a452a21e9c77d9dfa4828e3f012debf7784b19680ef587c\": container with ID starting with c9b724a3230411c85a452a21e9c77d9dfa4828e3f012debf7784b19680ef587c not found: ID does not exist"
Jul 14 16:01:22.064801 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.064775    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"]
Jul 14 16:01:22.067046 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.067026    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0e29de-openshift-default-6b9d85d7bc-vdx2n"]
Jul 14 16:01:22.218022 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:22.217989    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c206673-377b-4f43-bfb8-1009c989ba66" path="/var/lib/kubelet/pods/0c206673-377b-4f43-bfb8-1009c989ba66/volumes"
Jul 14 16:01:25.174182 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.174127    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"]
Jul 14 16:01:25.174659 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.174641    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c206673-377b-4f43-bfb8-1009c989ba66" containerName="istio-proxy"
Jul 14 16:01:25.174706 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.174661    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c206673-377b-4f43-bfb8-1009c989ba66" containerName="istio-proxy"
Jul 14 16:01:25.174760 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.174751    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c206673-377b-4f43-bfb8-1009c989ba66" containerName="istio-proxy"
Jul 14 16:01:25.174796 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.174764    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c206673-377b-4f43-bfb8-1009c989ba66" containerName="istio-proxy"
Jul 14 16:01:25.179114 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.179098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.181910 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.181890    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-519447-openshift-default-dockercfg-7h5qz\""
Jul 14 16:01:25.194221 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.194193    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"]
Jul 14 16:01:25.305434 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.305396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-workload-socket\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.305434 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.305434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-workload-certs\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.305810 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.305544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-istio-data\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.305810 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.305635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c94666-30d7-41e3-a345-882cd312cffd-istio-podinfo\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.305810 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.305755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-istio-envoy\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.305810 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.305788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-credential-socket\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.305959 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.305867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c94666-30d7-41e3-a345-882cd312cffd-istiod-ca-cert\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.305959 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.305891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8qwv\" (UniqueName: \"kubernetes.io/projected/39c94666-30d7-41e3-a345-882cd312cffd-kube-api-access-k8qwv\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.305959 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.305920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c94666-30d7-41e3-a345-882cd312cffd-istio-token\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.406785 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.406754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c94666-30d7-41e3-a345-882cd312cffd-istio-token\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.406956 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.406841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-workload-socket\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.406956 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.406868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-workload-certs\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.406956 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.406909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-istio-data\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.406956 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.406939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c94666-30d7-41e3-a345-882cd312cffd-istio-podinfo\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.407161 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.407002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-istio-envoy\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.407161 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.407025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-credential-socket\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.407161 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.407085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c94666-30d7-41e3-a345-882cd312cffd-istiod-ca-cert\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.407161 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.407110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8qwv\" (UniqueName: \"kubernetes.io/projected/39c94666-30d7-41e3-a345-882cd312cffd-kube-api-access-k8qwv\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.407375 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.407276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-workload-certs\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.407375 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.407330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-workload-socket\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.407573 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.407555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-credential-socket\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.407655 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.407571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-istio-data\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.407913 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.407859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c94666-30d7-41e3-a345-882cd312cffd-istiod-ca-cert\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.409433 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.409400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-istio-envoy\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.409716 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.409692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c94666-30d7-41e3-a345-882cd312cffd-istio-podinfo\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.416049 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.416028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c94666-30d7-41e3-a345-882cd312cffd-istio-token\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.416149 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.416105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8qwv\" (UniqueName: \"kubernetes.io/projected/39c94666-30d7-41e3-a345-882cd312cffd-kube-api-access-k8qwv\") pod \"e2e-shared-b-519447-openshift-default-768bfdff67-d7l54\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") " pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.491372 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.491282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:25.640222 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.640180    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"]
Jul 14 16:01:25.641687 ip-10-0-136-210 kubenswrapper[2563]: W0714 16:01:25.641651    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c94666_30d7_41e3_a345_882cd312cffd.slice/crio-6be8d2a4a89467255a547c7d82b34a7698c8b281595ca9ac6c458acfdcf55a2a WatchSource:0}: Error finding container 6be8d2a4a89467255a547c7d82b34a7698c8b281595ca9ac6c458acfdcf55a2a: Status 404 returned error can't find the container with id 6be8d2a4a89467255a547c7d82b34a7698c8b281595ca9ac6c458acfdcf55a2a
Jul 14 16:01:25.644075 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.644041    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:01:25.644151 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.644113    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:01:25.644151 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:25.644144    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:01:26.055557 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:26.055519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54" event={"ID":"39c94666-30d7-41e3-a345-882cd312cffd","Type":"ContainerStarted","Data":"79039d7cf1872acb021f1655a48cd3c34208024c72aa7c06829f40e792b3fe08"}
Jul 14 16:01:26.055744 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:26.055571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54" event={"ID":"39c94666-30d7-41e3-a345-882cd312cffd","Type":"ContainerStarted","Data":"6be8d2a4a89467255a547c7d82b34a7698c8b281595ca9ac6c458acfdcf55a2a"}
Jul 14 16:01:26.078028 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:26.077976    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54" podStartSLOduration=1.077959316 podStartE2EDuration="1.077959316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:01:26.077273798 +0000 UTC m=+2410.418539725" watchObservedRunningTime="2026-07-14 16:01:26.077959316 +0000 UTC m=+2410.419225222"
Jul 14 16:01:26.491881 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:26.491839    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:26.496775 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:26.496748    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:27.060878 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:27.060847    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:27.062028 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:27.062007    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:28.047182 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.047143    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:01:28.145123 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.145086    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"]
Jul 14 16:01:28.155245 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.155206    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c206673-377b-4f43-bfb8-1009c989ba66" containerName="istio-proxy"
Jul 14 16:01:28.155496 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.155480    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c206673-377b-4f43-bfb8-1009c989ba66" containerName="istio-proxy"
Jul 14 16:01:28.159838 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.159807    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"]
Jul 14 16:01:28.159984 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.159929    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:01:28.163918 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.163890    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-519447-serving-cert\""
Jul 14 16:01:28.245522 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.245463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0139426-42ac-452e-896b-6250023d5d23-maas-api-tls\") pod \"maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8\" (UID: \"b0139426-42ac-452e-896b-6250023d5d23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:01:28.245828 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.245803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0139426-42ac-452e-896b-6250023d5d23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8\" (UID: \"b0139426-42ac-452e-896b-6250023d5d23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:01:28.246020 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.246002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvsv\" (UniqueName: \"kubernetes.io/projected/b0139426-42ac-452e-896b-6250023d5d23-kube-api-access-8gvsv\") pod \"maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8\" (UID: \"b0139426-42ac-452e-896b-6250023d5d23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:01:28.347217 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.347100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0139426-42ac-452e-896b-6250023d5d23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8\" (UID: \"b0139426-42ac-452e-896b-6250023d5d23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:01:28.347217 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.347180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvsv\" (UniqueName: \"kubernetes.io/projected/b0139426-42ac-452e-896b-6250023d5d23-kube-api-access-8gvsv\") pod \"maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8\" (UID: \"b0139426-42ac-452e-896b-6250023d5d23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:01:28.347463 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.347326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0139426-42ac-452e-896b-6250023d5d23-maas-api-tls\") pod \"maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8\" (UID: \"b0139426-42ac-452e-896b-6250023d5d23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:01:28.347463 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:01:28.347447    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-serving-cert: secret "maas-api-e2e-shared-b-519447-serving-cert" not found
Jul 14 16:01:28.347603 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:01:28.347545    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0139426-42ac-452e-896b-6250023d5d23-maas-api-tls podName:b0139426-42ac-452e-896b-6250023d5d23 nodeName:}" failed. No retries permitted until 2026-07-14 16:01:28.847489343 +0000 UTC m=+2413.188755230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0139426-42ac-452e-896b-6250023d5d23-maas-api-tls") pod "maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8" (UID: "b0139426-42ac-452e-896b-6250023d5d23") : secret "maas-api-e2e-shared-b-519447-serving-cert" not found
Jul 14 16:01:28.347875 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.347855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0139426-42ac-452e-896b-6250023d5d23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8\" (UID: \"b0139426-42ac-452e-896b-6250023d5d23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:01:28.359519 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.359478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvsv\" (UniqueName: \"kubernetes.io/projected/b0139426-42ac-452e-896b-6250023d5d23-kube-api-access-8gvsv\") pod \"maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8\" (UID: \"b0139426-42ac-452e-896b-6250023d5d23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:01:28.852411 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.852368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0139426-42ac-452e-896b-6250023d5d23-maas-api-tls\") pod \"maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8\" (UID: \"b0139426-42ac-452e-896b-6250023d5d23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:01:28.854987 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:28.854966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0139426-42ac-452e-896b-6250023d5d23-maas-api-tls\") pod \"maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8\" (UID: \"b0139426-42ac-452e-896b-6250023d5d23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:01:29.072268 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:29.072230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:01:29.249937 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:29.249124    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"]
Jul 14 16:01:29.250011 ip-10-0-136-210 kubenswrapper[2563]: W0714 16:01:29.249947    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0139426_42ac_452e_896b_6250023d5d23.slice/crio-dcc15f630965cf14f0690abdd9c8a32e5d59498128b48ee42c40a88da9d2dab2 WatchSource:0}: Error finding container dcc15f630965cf14f0690abdd9c8a32e5d59498128b48ee42c40a88da9d2dab2: Status 404 returned error can't find the container with id dcc15f630965cf14f0690abdd9c8a32e5d59498128b48ee42c40a88da9d2dab2
Jul 14 16:01:30.079380 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:30.079333    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8" event={"ID":"b0139426-42ac-452e-896b-6250023d5d23","Type":"ContainerStarted","Data":"dcc15f630965cf14f0690abdd9c8a32e5d59498128b48ee42c40a88da9d2dab2"}
Jul 14 16:01:34.010662 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:34.010626    2563 patch_prober.go:28] interesting pod/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Jul 14 16:01:34.011074 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:34.010678    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8" podUID="afd5f36b-93a6-456c-8d55-c785a4a98869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Jul 14 16:01:34.106695 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:34.106665    2563 generic.go:356] "Generic (PLEG): container finished" podID="afd5f36b-93a6-456c-8d55-c785a4a98869" containerID="89617fa662aabe89616dcad7c869b01733b8657106c944c54f76aed145ffabab" exitCode=0
Jul 14 16:01:34.106878 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:34.106745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8" event={"ID":"afd5f36b-93a6-456c-8d55-c785a4a98869","Type":"ContainerDied","Data":"89617fa662aabe89616dcad7c869b01733b8657106c944c54f76aed145ffabab"}
Jul 14 16:01:34.107172 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:34.107160    2563 scope.go:117] "RemoveContainer" containerID="89617fa662aabe89616dcad7c869b01733b8657106c944c54f76aed145ffabab"
Jul 14 16:01:34.108209 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:34.108184    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:01:34.108270 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:34.108248    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:01:34.108303 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:34.108285    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:01:35.113655 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:35.113622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8" event={"ID":"b0139426-42ac-452e-896b-6250023d5d23","Type":"ContainerStarted","Data":"651bb80dad1fa8c7fd3c4bde19dc25c0547eb40e79aa9111eff97eecd055e576"}
Jul 14 16:01:35.114099 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:35.113712    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:01:35.115596 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:35.115574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8" event={"ID":"afd5f36b-93a6-456c-8d55-c785a4a98869","Type":"ContainerStarted","Data":"ae5bf17bf77bab70d1b03378346ee8f1f3b6790b869aeaa051006e64d04f3beb"}
Jul 14 16:01:35.158693 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:35.158614    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8" podStartSLOduration=1.428127996 podStartE2EDuration="7.15857702s" totalImagesPullingTime="5.730449024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 16:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:01:35.136733837 +0000 UTC m=+2419.477999762" watchObservedRunningTime="2026-07-14 16:01:35.15857702 +0000 UTC m=+2419.499842928"
Jul 14 16:01:35.688715 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:35.688676    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:35.689140 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:35.689111    2563 patch_prober.go:28] interesting pod/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Jul 14 16:01:35.689268 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:35.689191    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8" podUID="afd5f36b-93a6-456c-8d55-c785a4a98869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Jul 14 16:01:36.688930 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:36.688843    2563 patch_prober.go:28] interesting pod/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Jul 14 16:01:36.688930 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:36.688895    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8" podUID="afd5f36b-93a6-456c-8d55-c785a4a98869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Jul 14 16:01:37.689862 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:37.689822    2563 patch_prober.go:28] interesting pod/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Jul 14 16:01:37.690291 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:37.689881    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8" podUID="afd5f36b-93a6-456c-8d55-c785a4a98869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Jul 14 16:01:38.689390 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:38.689349    2563 patch_prober.go:28] interesting pod/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Jul 14 16:01:38.689577 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:38.689415    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8" podUID="afd5f36b-93a6-456c-8d55-c785a4a98869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Jul 14 16:01:39.694420 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:39.694386    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:39.694928 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:39.694761    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:39.695546 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:39.695525    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:01:41.131096 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:41.131070    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:01:42.062232 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:42.062196    2563 patch_prober.go:28] interesting pod/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Jul 14 16:01:42.062436 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:42.062260    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54" podUID="39c94666-30d7-41e3-a345-882cd312cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 14 16:01:42.185402 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:42.185369    2563 generic.go:356] "Generic (PLEG): container finished" podID="39c94666-30d7-41e3-a345-882cd312cffd" containerID="79039d7cf1872acb021f1655a48cd3c34208024c72aa7c06829f40e792b3fe08" exitCode=0
Jul 14 16:01:42.185804 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:42.185448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54" event={"ID":"39c94666-30d7-41e3-a345-882cd312cffd","Type":"ContainerDied","Data":"79039d7cf1872acb021f1655a48cd3c34208024c72aa7c06829f40e792b3fe08"}
Jul 14 16:01:42.185913 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:42.185898    2563 scope.go:117] "RemoveContainer" containerID="79039d7cf1872acb021f1655a48cd3c34208024c72aa7c06829f40e792b3fe08"
Jul 14 16:01:42.186930 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:42.186903    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:01:42.186997 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:42.186974    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:01:42.187039 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:42.187004    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Jul 14 16:01:43.193045 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:43.193008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54" event={"ID":"39c94666-30d7-41e3-a345-882cd312cffd","Type":"ContainerStarted","Data":"741f1231ef6061234671631187422d5c0acb562f165d9371233de1dc0f8701f8"}
Jul 14 16:01:43.492226 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:43.492138    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:43.492374 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:43.492340    2563 patch_prober.go:28] interesting pod/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Jul 14 16:01:43.492416 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:43.492396    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54" podUID="39c94666-30d7-41e3-a345-882cd312cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 14 16:01:44.491753 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:44.491715    2563 patch_prober.go:28] interesting pod/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Jul 14 16:01:44.492224 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:44.491767    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54" podUID="39c94666-30d7-41e3-a345-882cd312cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 14 16:01:45.492520 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:45.492478    2563 patch_prober.go:28] interesting pod/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Jul 14 16:01:45.492896 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:45.492544    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54" podUID="39c94666-30d7-41e3-a345-882cd312cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 14 16:01:46.492425 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:46.492383    2563 patch_prober.go:28] interesting pod/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Jul 14 16:01:46.492643 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:46.492468    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54" podUID="39c94666-30d7-41e3-a345-882cd312cffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 14 16:01:47.495874 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:47.495843    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:47.496277 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:47.496201    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:01:47.497007 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:01:47.496985    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:06:16.330826 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:06:16.330742    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 16:06:16.334784 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:06:16.334763    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jfnl2_7e71167e-01b8-42b0-9403-8ae910ec77bf/ovn-acl-logging/0.log"
Jul 14 16:08:12.099317 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.099276    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"]
Jul 14 16:08:12.099874 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.099545    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88" podUID="b5a50bda-98a9-4058-ab8a-fd1d16df1887" containerName="maas-api" containerID="cri-o://3c8dc4539e0efd7223bd5ae8c086ad87cc40868a1aa4e5bdf7c6b93ae7e2696a" gracePeriod=30
Jul 14 16:08:12.356910 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.356850    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:08:12.408427 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.408392    2563 generic.go:356] "Generic (PLEG): container finished" podID="b5a50bda-98a9-4058-ab8a-fd1d16df1887" containerID="3c8dc4539e0efd7223bd5ae8c086ad87cc40868a1aa4e5bdf7c6b93ae7e2696a" exitCode=0
Jul 14 16:08:12.408614 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.408434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88" event={"ID":"b5a50bda-98a9-4058-ab8a-fd1d16df1887","Type":"ContainerDied","Data":"3c8dc4539e0efd7223bd5ae8c086ad87cc40868a1aa4e5bdf7c6b93ae7e2696a"}
Jul 14 16:08:12.408614 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.408488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88" event={"ID":"b5a50bda-98a9-4058-ab8a-fd1d16df1887","Type":"ContainerDied","Data":"8d187d39ec4391619670ecf9ecd294ce6dff24861362123ce9b34c83b769d984"}
Jul 14 16:08:12.408614 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.408443    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"
Jul 14 16:08:12.408614 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.408533    2563 scope.go:117] "RemoveContainer" containerID="3c8dc4539e0efd7223bd5ae8c086ad87cc40868a1aa4e5bdf7c6b93ae7e2696a"
Jul 14 16:08:12.418386 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.418361    2563 scope.go:117] "RemoveContainer" containerID="3c8dc4539e0efd7223bd5ae8c086ad87cc40868a1aa4e5bdf7c6b93ae7e2696a"
Jul 14 16:08:12.418661 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:08:12.418641    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8dc4539e0efd7223bd5ae8c086ad87cc40868a1aa4e5bdf7c6b93ae7e2696a\": container with ID starting with 3c8dc4539e0efd7223bd5ae8c086ad87cc40868a1aa4e5bdf7c6b93ae7e2696a not found: ID does not exist" containerID="3c8dc4539e0efd7223bd5ae8c086ad87cc40868a1aa4e5bdf7c6b93ae7e2696a"
Jul 14 16:08:12.418736 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.418675    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8dc4539e0efd7223bd5ae8c086ad87cc40868a1aa4e5bdf7c6b93ae7e2696a"} err="failed to get container status \"3c8dc4539e0efd7223bd5ae8c086ad87cc40868a1aa4e5bdf7c6b93ae7e2696a\": rpc error: code = NotFound desc = could not find container \"3c8dc4539e0efd7223bd5ae8c086ad87cc40868a1aa4e5bdf7c6b93ae7e2696a\": container with ID starting with 3c8dc4539e0efd7223bd5ae8c086ad87cc40868a1aa4e5bdf7c6b93ae7e2696a not found: ID does not exist"
Jul 14 16:08:12.453617 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.453591    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a50bda-98a9-4058-ab8a-fd1d16df1887-maas-api-tls\") pod \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\" (UID: \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\") "
Jul 14 16:08:12.453780 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.453638    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a50bda-98a9-4058-ab8a-fd1d16df1887-openshift-service-ca\") pod \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\" (UID: \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\") "
Jul 14 16:08:12.453780 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.453686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nhf8\" (UniqueName: \"kubernetes.io/projected/b5a50bda-98a9-4058-ab8a-fd1d16df1887-kube-api-access-8nhf8\") pod \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\" (UID: \"b5a50bda-98a9-4058-ab8a-fd1d16df1887\") "
Jul 14 16:08:12.454072 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.454047    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a50bda-98a9-4058-ab8a-fd1d16df1887-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b5a50bda-98a9-4058-ab8a-fd1d16df1887" (UID: "b5a50bda-98a9-4058-ab8a-fd1d16df1887"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 16:08:12.455756 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.455726    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a50bda-98a9-4058-ab8a-fd1d16df1887-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b5a50bda-98a9-4058-ab8a-fd1d16df1887" (UID: "b5a50bda-98a9-4058-ab8a-fd1d16df1887"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 16:08:12.455862 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.455829    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a50bda-98a9-4058-ab8a-fd1d16df1887-kube-api-access-8nhf8" (OuterVolumeSpecName: "kube-api-access-8nhf8") pod "b5a50bda-98a9-4058-ab8a-fd1d16df1887" (UID: "b5a50bda-98a9-4058-ab8a-fd1d16df1887"). InnerVolumeSpecName "kube-api-access-8nhf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 16:08:12.554394 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.554365    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a50bda-98a9-4058-ab8a-fd1d16df1887-maas-api-tls\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:12.554394 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.554391    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a50bda-98a9-4058-ab8a-fd1d16df1887-openshift-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:12.554394 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.554403    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nhf8\" (UniqueName: \"kubernetes.io/projected/b5a50bda-98a9-4058-ab8a-fd1d16df1887-kube-api-access-8nhf8\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:12.736827 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.736794    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"]
Jul 14 16:08:12.740892 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:12.740869    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bea800-86f7466bb9-spt88"]
Jul 14 16:08:14.216281 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:14.216253    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5a50bda-98a9-4058-ab8a-fd1d16df1887" path="/var/lib/kubelet/pods/b5a50bda-98a9-4058-ab8a-fd1d16df1887/volumes"
Jul 14 16:08:27.418378 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:27.418347    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"]
Jul 14 16:08:27.418776 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:27.418599    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8" podUID="afd5f36b-93a6-456c-8d55-c785a4a98869" containerName="istio-proxy" containerID="cri-o://ae5bf17bf77bab70d1b03378346ee8f1f3b6790b869aeaa051006e64d04f3beb" gracePeriod=30
Jul 14 16:08:32.680644 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.680585    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:08:32.756712 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.756680    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-data\") pod \"afd5f36b-93a6-456c-8d55-c785a4a98869\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") "
Jul 14 16:08:32.756878 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.756737    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-workload-certs\") pod \"afd5f36b-93a6-456c-8d55-c785a4a98869\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") "
Jul 14 16:08:32.756878 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.756805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-envoy\") pod \"afd5f36b-93a6-456c-8d55-c785a4a98869\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") "
Jul 14 16:08:32.756878 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.756838    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-podinfo\") pod \"afd5f36b-93a6-456c-8d55-c785a4a98869\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") "
Jul 14 16:08:32.756878 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.756864    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-token\") pod \"afd5f36b-93a6-456c-8d55-c785a4a98869\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") "
Jul 14 16:08:32.757090 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.756978    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-credential-socket\") pod \"afd5f36b-93a6-456c-8d55-c785a4a98869\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") "
Jul 14 16:08:32.757090 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.757044    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "afd5f36b-93a6-456c-8d55-c785a4a98869" (UID: "afd5f36b-93a6-456c-8d55-c785a4a98869"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:08:32.757283 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.757047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj5ws\" (UniqueName: \"kubernetes.io/projected/afd5f36b-93a6-456c-8d55-c785a4a98869-kube-api-access-wj5ws\") pod \"afd5f36b-93a6-456c-8d55-c785a4a98869\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") "
Jul 14 16:08:32.757625 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.757596    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "afd5f36b-93a6-456c-8d55-c785a4a98869" (UID: "afd5f36b-93a6-456c-8d55-c785a4a98869"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:08:32.757625 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.757605    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd5f36b-93a6-456c-8d55-c785a4a98869-istiod-ca-cert\") pod \"afd5f36b-93a6-456c-8d55-c785a4a98869\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") "
Jul 14 16:08:32.757803 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.757655    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-data" (OuterVolumeSpecName: "istio-data") pod "afd5f36b-93a6-456c-8d55-c785a4a98869" (UID: "afd5f36b-93a6-456c-8d55-c785a4a98869"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:08:32.757803 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.757674    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-workload-socket\") pod \"afd5f36b-93a6-456c-8d55-c785a4a98869\" (UID: \"afd5f36b-93a6-456c-8d55-c785a4a98869\") "
Jul 14 16:08:32.757929 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.757909    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd5f36b-93a6-456c-8d55-c785a4a98869-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "afd5f36b-93a6-456c-8d55-c785a4a98869" (UID: "afd5f36b-93a6-456c-8d55-c785a4a98869"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 16:08:32.758163 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.758074    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "afd5f36b-93a6-456c-8d55-c785a4a98869" (UID: "afd5f36b-93a6-456c-8d55-c785a4a98869"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:08:32.758836 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.758807    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-data\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:32.758967 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.758840    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-workload-certs\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:32.758967 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.758855    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-credential-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:32.758967 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.758879    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd5f36b-93a6-456c-8d55-c785a4a98869-istiod-ca-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:32.758967 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.758893    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-workload-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:32.760841 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.760817    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "afd5f36b-93a6-456c-8d55-c785a4a98869" (UID: "afd5f36b-93a6-456c-8d55-c785a4a98869"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 16:08:32.760980 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.760840    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd5f36b-93a6-456c-8d55-c785a4a98869-kube-api-access-wj5ws" (OuterVolumeSpecName: "kube-api-access-wj5ws") pod "afd5f36b-93a6-456c-8d55-c785a4a98869" (UID: "afd5f36b-93a6-456c-8d55-c785a4a98869"). InnerVolumeSpecName "kube-api-access-wj5ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 16:08:32.761072 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.760915    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-token" (OuterVolumeSpecName: "istio-token") pod "afd5f36b-93a6-456c-8d55-c785a4a98869" (UID: "afd5f36b-93a6-456c-8d55-c785a4a98869"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 16:08:32.767548 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.767495    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "afd5f36b-93a6-456c-8d55-c785a4a98869" (UID: "afd5f36b-93a6-456c-8d55-c785a4a98869"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:08:32.859587 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.859549    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj5ws\" (UniqueName: \"kubernetes.io/projected/afd5f36b-93a6-456c-8d55-c785a4a98869-kube-api-access-wj5ws\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:32.859587 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.859580    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-envoy\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:32.859587 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.859590    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-podinfo\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:32.859587 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:32.859598    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd5f36b-93a6-456c-8d55-c785a4a98869-istio-token\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:33.530387 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:33.530354    2563 generic.go:356] "Generic (PLEG): container finished" podID="afd5f36b-93a6-456c-8d55-c785a4a98869" containerID="ae5bf17bf77bab70d1b03378346ee8f1f3b6790b869aeaa051006e64d04f3beb" exitCode=0
Jul 14 16:08:33.530577 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:33.530411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8" event={"ID":"afd5f36b-93a6-456c-8d55-c785a4a98869","Type":"ContainerDied","Data":"ae5bf17bf77bab70d1b03378346ee8f1f3b6790b869aeaa051006e64d04f3beb"}
Jul 14 16:08:33.530577 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:33.530421    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"
Jul 14 16:08:33.530577 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:33.530452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8" event={"ID":"afd5f36b-93a6-456c-8d55-c785a4a98869","Type":"ContainerDied","Data":"30e941eb90ef8d599377c0d63ab80b979d5186c7ef46621f8aaa38014b77d92b"}
Jul 14 16:08:33.530577 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:33.530474    2563 scope.go:117] "RemoveContainer" containerID="ae5bf17bf77bab70d1b03378346ee8f1f3b6790b869aeaa051006e64d04f3beb"
Jul 14 16:08:33.540977 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:33.540769    2563 scope.go:117] "RemoveContainer" containerID="89617fa662aabe89616dcad7c869b01733b8657106c944c54f76aed145ffabab"
Jul 14 16:08:33.549231 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:33.549211    2563 scope.go:117] "RemoveContainer" containerID="ae5bf17bf77bab70d1b03378346ee8f1f3b6790b869aeaa051006e64d04f3beb"
Jul 14 16:08:33.549470 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:08:33.549451    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5bf17bf77bab70d1b03378346ee8f1f3b6790b869aeaa051006e64d04f3beb\": container with ID starting with ae5bf17bf77bab70d1b03378346ee8f1f3b6790b869aeaa051006e64d04f3beb not found: ID does not exist" containerID="ae5bf17bf77bab70d1b03378346ee8f1f3b6790b869aeaa051006e64d04f3beb"
Jul 14 16:08:33.549532 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:33.549481    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5bf17bf77bab70d1b03378346ee8f1f3b6790b869aeaa051006e64d04f3beb"} err="failed to get container status \"ae5bf17bf77bab70d1b03378346ee8f1f3b6790b869aeaa051006e64d04f3beb\": rpc error: code = NotFound desc = could not find container \"ae5bf17bf77bab70d1b03378346ee8f1f3b6790b869aeaa051006e64d04f3beb\": container with ID starting with ae5bf17bf77bab70d1b03378346ee8f1f3b6790b869aeaa051006e64d04f3beb not found: ID does not exist"
Jul 14 16:08:33.549532 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:33.549499    2563 scope.go:117] "RemoveContainer" containerID="89617fa662aabe89616dcad7c869b01733b8657106c944c54f76aed145ffabab"
Jul 14 16:08:33.549734 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:08:33.549715    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89617fa662aabe89616dcad7c869b01733b8657106c944c54f76aed145ffabab\": container with ID starting with 89617fa662aabe89616dcad7c869b01733b8657106c944c54f76aed145ffabab not found: ID does not exist" containerID="89617fa662aabe89616dcad7c869b01733b8657106c944c54f76aed145ffabab"
Jul 14 16:08:33.549773 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:33.549740    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89617fa662aabe89616dcad7c869b01733b8657106c944c54f76aed145ffabab"} err="failed to get container status \"89617fa662aabe89616dcad7c869b01733b8657106c944c54f76aed145ffabab\": rpc error: code = NotFound desc = could not find container \"89617fa662aabe89616dcad7c869b01733b8657106c944c54f76aed145ffabab\": container with ID starting with 89617fa662aabe89616dcad7c869b01733b8657106c944c54f76aed145ffabab not found: ID does not exist"
Jul 14 16:08:33.556334 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:33.556312    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"]
Jul 14 16:08:33.561083 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:33.561061    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-bea800-openshift-default-6cc9546bc5-sdbk8"]
Jul 14 16:08:34.216686 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:34.216654    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afd5f36b-93a6-456c-8d55-c785a4a98869" path="/var/lib/kubelet/pods/afd5f36b-93a6-456c-8d55-c785a4a98869/volumes"
Jul 14 16:08:38.375227 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:38.375193    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"]
Jul 14 16:08:38.375690 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:38.375399    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8" podUID="b0139426-42ac-452e-896b-6250023d5d23" containerName="maas-api" containerID="cri-o://651bb80dad1fa8c7fd3c4bde19dc25c0547eb40e79aa9111eff97eecd055e576" gracePeriod=30
Jul 14 16:08:38.560905 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:38.560876    2563 generic.go:356] "Generic (PLEG): container finished" podID="b0139426-42ac-452e-896b-6250023d5d23" containerID="651bb80dad1fa8c7fd3c4bde19dc25c0547eb40e79aa9111eff97eecd055e576" exitCode=0
Jul 14 16:08:38.561060 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:38.560908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8" event={"ID":"b0139426-42ac-452e-896b-6250023d5d23","Type":"ContainerDied","Data":"651bb80dad1fa8c7fd3c4bde19dc25c0547eb40e79aa9111eff97eecd055e576"}
Jul 14 16:08:38.627704 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:38.627637    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:08:38.818788 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:38.818759    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0139426-42ac-452e-896b-6250023d5d23-maas-api-tls\") pod \"b0139426-42ac-452e-896b-6250023d5d23\" (UID: \"b0139426-42ac-452e-896b-6250023d5d23\") "
Jul 14 16:08:38.818959 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:38.818874    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gvsv\" (UniqueName: \"kubernetes.io/projected/b0139426-42ac-452e-896b-6250023d5d23-kube-api-access-8gvsv\") pod \"b0139426-42ac-452e-896b-6250023d5d23\" (UID: \"b0139426-42ac-452e-896b-6250023d5d23\") "
Jul 14 16:08:38.818959 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:38.818941    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0139426-42ac-452e-896b-6250023d5d23-openshift-service-ca\") pod \"b0139426-42ac-452e-896b-6250023d5d23\" (UID: \"b0139426-42ac-452e-896b-6250023d5d23\") "
Jul 14 16:08:38.819331 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:38.819305    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0139426-42ac-452e-896b-6250023d5d23-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0139426-42ac-452e-896b-6250023d5d23" (UID: "b0139426-42ac-452e-896b-6250023d5d23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 16:08:38.820978 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:38.820939    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0139426-42ac-452e-896b-6250023d5d23-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0139426-42ac-452e-896b-6250023d5d23" (UID: "b0139426-42ac-452e-896b-6250023d5d23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 16:08:38.821088 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:38.821064    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0139426-42ac-452e-896b-6250023d5d23-kube-api-access-8gvsv" (OuterVolumeSpecName: "kube-api-access-8gvsv") pod "b0139426-42ac-452e-896b-6250023d5d23" (UID: "b0139426-42ac-452e-896b-6250023d5d23"). InnerVolumeSpecName "kube-api-access-8gvsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 16:08:38.920597 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:38.920554    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gvsv\" (UniqueName: \"kubernetes.io/projected/b0139426-42ac-452e-896b-6250023d5d23-kube-api-access-8gvsv\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:38.920597 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:38.920596    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0139426-42ac-452e-896b-6250023d5d23-openshift-service-ca\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:38.920796 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:38.920612    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0139426-42ac-452e-896b-6250023d5d23-maas-api-tls\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:39.567683 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:39.567654    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"
Jul 14 16:08:39.567683 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:39.567667    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8" event={"ID":"b0139426-42ac-452e-896b-6250023d5d23","Type":"ContainerDied","Data":"dcc15f630965cf14f0690abdd9c8a32e5d59498128b48ee42c40a88da9d2dab2"}
Jul 14 16:08:39.568186 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:39.567722    2563 scope.go:117] "RemoveContainer" containerID="651bb80dad1fa8c7fd3c4bde19dc25c0547eb40e79aa9111eff97eecd055e576"
Jul 14 16:08:39.591049 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:39.591024    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"]
Jul 14 16:08:39.595615 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:39.595590    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-519447-6bcb8856d4-zpkw8"]
Jul 14 16:08:40.217178 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:40.217150    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0139426-42ac-452e-896b-6250023d5d23" path="/var/lib/kubelet/pods/b0139426-42ac-452e-896b-6250023d5d23/volumes"
Jul 14 16:08:53.638878 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:53.638835    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"]
Jul 14 16:08:53.639282 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:53.639049    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54" podUID="39c94666-30d7-41e3-a345-882cd312cffd" containerName="istio-proxy" containerID="cri-o://741f1231ef6061234671631187422d5c0acb562f165d9371233de1dc0f8701f8" gracePeriod=30
Jul 14 16:08:58.894339 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:58.894312    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:08:59.017352 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.017247    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c94666-30d7-41e3-a345-882cd312cffd-istio-token\") pod \"39c94666-30d7-41e3-a345-882cd312cffd\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") "
Jul 14 16:08:59.017352 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.017298    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8qwv\" (UniqueName: \"kubernetes.io/projected/39c94666-30d7-41e3-a345-882cd312cffd-kube-api-access-k8qwv\") pod \"39c94666-30d7-41e3-a345-882cd312cffd\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") "
Jul 14 16:08:59.017352 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.017352    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-workload-certs\") pod \"39c94666-30d7-41e3-a345-882cd312cffd\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") "
Jul 14 16:08:59.017678 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.017402    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c94666-30d7-41e3-a345-882cd312cffd-istio-podinfo\") pod \"39c94666-30d7-41e3-a345-882cd312cffd\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") "
Jul 14 16:08:59.017678 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.017432    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-istio-data\") pod \"39c94666-30d7-41e3-a345-882cd312cffd\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") "
Jul 14 16:08:59.017678 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.017464    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-credential-socket\") pod \"39c94666-30d7-41e3-a345-882cd312cffd\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") "
Jul 14 16:08:59.017678 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.017557    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c94666-30d7-41e3-a345-882cd312cffd-istiod-ca-cert\") pod \"39c94666-30d7-41e3-a345-882cd312cffd\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") "
Jul 14 16:08:59.017678 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.017619    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-istio-envoy\") pod \"39c94666-30d7-41e3-a345-882cd312cffd\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") "
Jul 14 16:08:59.017678 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.017650    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-workload-socket\") pod \"39c94666-30d7-41e3-a345-882cd312cffd\" (UID: \"39c94666-30d7-41e3-a345-882cd312cffd\") "
Jul 14 16:08:59.017983 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.017723    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39c94666-30d7-41e3-a345-882cd312cffd" (UID: "39c94666-30d7-41e3-a345-882cd312cffd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:08:59.017983 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.017803    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39c94666-30d7-41e3-a345-882cd312cffd" (UID: "39c94666-30d7-41e3-a345-882cd312cffd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:08:59.018081 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.017989    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39c94666-30d7-41e3-a345-882cd312cffd" (UID: "39c94666-30d7-41e3-a345-882cd312cffd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:08:59.018081 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.018042    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c94666-30d7-41e3-a345-882cd312cffd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39c94666-30d7-41e3-a345-882cd312cffd" (UID: "39c94666-30d7-41e3-a345-882cd312cffd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 16:08:59.018194 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.018119    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-workload-certs\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:59.018194 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.018138    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-credential-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:59.018194 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.018152    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-workload-socket\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:59.018570 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.018554    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-istio-data" (OuterVolumeSpecName: "istio-data") pod "39c94666-30d7-41e3-a345-882cd312cffd" (UID: "39c94666-30d7-41e3-a345-882cd312cffd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:08:59.020206 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.020184    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c94666-30d7-41e3-a345-882cd312cffd-kube-api-access-k8qwv" (OuterVolumeSpecName: "kube-api-access-k8qwv") pod "39c94666-30d7-41e3-a345-882cd312cffd" (UID: "39c94666-30d7-41e3-a345-882cd312cffd"). InnerVolumeSpecName "kube-api-access-k8qwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 16:08:59.020206 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.020182    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c94666-30d7-41e3-a345-882cd312cffd-istio-token" (OuterVolumeSpecName: "istio-token") pod "39c94666-30d7-41e3-a345-882cd312cffd" (UID: "39c94666-30d7-41e3-a345-882cd312cffd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 16:08:59.020379 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.020314    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39c94666-30d7-41e3-a345-882cd312cffd" (UID: "39c94666-30d7-41e3-a345-882cd312cffd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 16:08:59.020437 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.020416    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39c94666-30d7-41e3-a345-882cd312cffd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39c94666-30d7-41e3-a345-882cd312cffd" (UID: "39c94666-30d7-41e3-a345-882cd312cffd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 16:08:59.119382 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.119354    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c94666-30d7-41e3-a345-882cd312cffd-istio-token\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:59.119382 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.119378    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8qwv\" (UniqueName: \"kubernetes.io/projected/39c94666-30d7-41e3-a345-882cd312cffd-kube-api-access-k8qwv\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:59.119382 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.119389    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c94666-30d7-41e3-a345-882cd312cffd-istio-podinfo\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:59.119630 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.119397    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-istio-data\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:59.119630 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.119406    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c94666-30d7-41e3-a345-882cd312cffd-istiod-ca-cert\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:59.119630 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.119414    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c94666-30d7-41e3-a345-882cd312cffd-istio-envoy\") on node \"ip-10-0-136-210.ec2.internal\" DevicePath \"\""
Jul 14 16:08:59.649780 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.649743    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-wg277_0ccbedea-ad39-458c-89af-4fbb1801359a/manager/0.log"
Jul 14 16:08:59.690311 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.690270    2563 generic.go:356] "Generic (PLEG): container finished" podID="39c94666-30d7-41e3-a345-882cd312cffd" containerID="741f1231ef6061234671631187422d5c0acb562f165d9371233de1dc0f8701f8" exitCode=0
Jul 14 16:08:59.690473 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.690329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54" event={"ID":"39c94666-30d7-41e3-a345-882cd312cffd","Type":"ContainerDied","Data":"741f1231ef6061234671631187422d5c0acb562f165d9371233de1dc0f8701f8"}
Jul 14 16:08:59.690473 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.690366    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"
Jul 14 16:08:59.690473 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.690411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54" event={"ID":"39c94666-30d7-41e3-a345-882cd312cffd","Type":"ContainerDied","Data":"6be8d2a4a89467255a547c7d82b34a7698c8b281595ca9ac6c458acfdcf55a2a"}
Jul 14 16:08:59.690473 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.690434    2563 scope.go:117] "RemoveContainer" containerID="741f1231ef6061234671631187422d5c0acb562f165d9371233de1dc0f8701f8"
Jul 14 16:08:59.700349 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.700331    2563 scope.go:117] "RemoveContainer" containerID="79039d7cf1872acb021f1655a48cd3c34208024c72aa7c06829f40e792b3fe08"
Jul 14 16:08:59.708691 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.708670    2563 scope.go:117] "RemoveContainer" containerID="741f1231ef6061234671631187422d5c0acb562f165d9371233de1dc0f8701f8"
Jul 14 16:08:59.708919 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:08:59.708900    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741f1231ef6061234671631187422d5c0acb562f165d9371233de1dc0f8701f8\": container with ID starting with 741f1231ef6061234671631187422d5c0acb562f165d9371233de1dc0f8701f8 not found: ID does not exist" containerID="741f1231ef6061234671631187422d5c0acb562f165d9371233de1dc0f8701f8"
Jul 14 16:08:59.708989 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.708931    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741f1231ef6061234671631187422d5c0acb562f165d9371233de1dc0f8701f8"} err="failed to get container status \"741f1231ef6061234671631187422d5c0acb562f165d9371233de1dc0f8701f8\": rpc error: code = NotFound desc = could not find container \"741f1231ef6061234671631187422d5c0acb562f165d9371233de1dc0f8701f8\": container with ID starting with 741f1231ef6061234671631187422d5c0acb562f165d9371233de1dc0f8701f8 not found: ID does not exist"
Jul 14 16:08:59.708989 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.708955    2563 scope.go:117] "RemoveContainer" containerID="79039d7cf1872acb021f1655a48cd3c34208024c72aa7c06829f40e792b3fe08"
Jul 14 16:08:59.709173 ip-10-0-136-210 kubenswrapper[2563]: E0714 16:08:59.709157    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79039d7cf1872acb021f1655a48cd3c34208024c72aa7c06829f40e792b3fe08\": container with ID starting with 79039d7cf1872acb021f1655a48cd3c34208024c72aa7c06829f40e792b3fe08 not found: ID does not exist" containerID="79039d7cf1872acb021f1655a48cd3c34208024c72aa7c06829f40e792b3fe08"
Jul 14 16:08:59.709213 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.709181    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79039d7cf1872acb021f1655a48cd3c34208024c72aa7c06829f40e792b3fe08"} err="failed to get container status \"79039d7cf1872acb021f1655a48cd3c34208024c72aa7c06829f40e792b3fe08\": rpc error: code = NotFound desc = could not find container \"79039d7cf1872acb021f1655a48cd3c34208024c72aa7c06829f40e792b3fe08\": container with ID starting with 79039d7cf1872acb021f1655a48cd3c34208024c72aa7c06829f40e792b3fe08 not found: ID does not exist"
Jul 14 16:08:59.715142 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.715118    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"]
Jul 14 16:08:59.720917 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.720895    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-519447-openshift-default-768bfdff67-d7l54"]
Jul 14 16:08:59.784915 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:08:59.784890    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wsnwc_1367dbde-4785-4189-a4e5-cb457c93c18b/manager/0.log"
Jul 14 16:09:00.216431 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:00.216403    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39c94666-30d7-41e3-a345-882cd312cffd" path="/var/lib/kubelet/pods/39c94666-30d7-41e3-a345-882cd312cffd/volumes"
Jul 14 16:09:00.436360 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:00.436335    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66c4b7dd84-7j48t_3e8f4479-2e26-46c8-9b6d-45f72af91d02/copy-manifests/0.log"
Jul 14 16:09:00.445014 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:00.444955    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66c4b7dd84-7j48t_3e8f4479-2e26-46c8-9b6d-45f72af91d02/manager/0.log"
Jul 14 16:09:01.060876 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:01.060852    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734065-qf89f_f9d5adbd-7d3e-43eb-b900-79f6eb11fc3b/cleanup/0.log"
Jul 14 16:09:01.179701 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:01.179667    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734080-2xqkr_39978635-b618-408c-84d1-8dc2107531e2/cleanup/0.log"
Jul 14 16:09:01.306313 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:01.306227    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-94af7c4d-q57kx_e79ea4b9-d059-44f2-894e-4b7b0785e4d5/revoke-keys/0.log"
Jul 14 16:09:01.443802 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:01.443775    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-cd6a0c0f-575jp_abf553a2-1dda-439e-b05f-aa6cc3ca6f34/revoke-keys/0.log"
Jul 14 16:09:01.787837 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:01.787805    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-390b960b-hbl49_523fa1a9-2d95-4a61-aa12-1c5060c86a20/revoke-keys/0.log"
Jul 14 16:09:01.901341 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:01.901307    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-eb3623-r455l_11074954-387e-4b6d-bb73-726d19215daf/revoke-keys/0.log"
Jul 14 16:09:02.140854 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:02.140821    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-65fd3bf5-gbsr7_06b8feca-eb12-4b86-9634-280d60af3171/revoke-keys/0.log"
Jul 14 16:09:02.265705 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:02.265677    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-5e35a406-sgsxz_b46b2efc-c45f-46bf-9a9e-2caa17acb588/revoke-keys/0.log"
Jul 14 16:09:02.390024 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:02.390001    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-9cd0814a-thm2r_66b9e4de-5f94-4ff6-bfa5-9abe071cc174/revoke-keys/0.log"
Jul 14 16:09:04.247475 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:04.247445    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kn2cz_f7655a49-edcc-4c31-9ecf-41f2ad8dd633/manager/0.log"
Jul 14 16:09:04.374183 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:04.374152    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ltthc_736c5c4a-eda7-4342-8971-ee6c7f0eff1d/manager/0.log"
Jul 14 16:09:04.619860 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:04.619778    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wvm95_81a4323c-6cfb-4f75-8091-9d4fc9d2ac70/registry-server/0.log"
Jul 14 16:09:05.005541 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:05.005493    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-z2pt5_8d763c04-1a1e-409a-9ced-aad811881e54/manager/0.log"
Jul 14 16:09:06.899695 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:06.899619    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-nfrct_bff7729f-90fd-4c99-b16d-f61454431316/main/0.log"
Jul 14 16:09:07.151836 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:07.151751    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-xgb9k_d6b4e6a6-b1b3-44fe-899f-49e8688542b9/main/0.log"
Jul 14 16:09:07.270344 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:07.270311    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-4hm44_40eff3ef-ea1d-411c-baea-c23b0ee3aeb9/main/0.log"
Jul 14 16:09:14.614889 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:14.614844    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w8dm9_d3ae120a-8040-483a-8da6-090ef1e80265/global-pull-secret-syncer/0.log"
Jul 14 16:09:14.768474 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:14.768449    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rtlt8_15614b5c-a711-470b-92b6-7ad03154f6be/konnectivity-agent/0.log"
Jul 14 16:09:14.850892 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:14.850859    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-210.ec2.internal_3d95b182bc9232c53452fc18d92789eb/haproxy/0.log"
Jul 14 16:09:19.546792 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:19.546705    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kn2cz_f7655a49-edcc-4c31-9ecf-41f2ad8dd633/manager/0.log"
Jul 14 16:09:19.599693 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:19.599662    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ltthc_736c5c4a-eda7-4342-8971-ee6c7f0eff1d/manager/0.log"
Jul 14 16:09:19.694058 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:19.694027    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wvm95_81a4323c-6cfb-4f75-8091-9d4fc9d2ac70/registry-server/0.log"
Jul 14 16:09:20.051672 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:20.051641    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-z2pt5_8d763c04-1a1e-409a-9ced-aad811881e54/manager/0.log"
Jul 14 16:09:21.903733 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:21.903703    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63ed7786-f31d-4538-a5d1-29b328cf5c23/alertmanager/0.log"
Jul 14 16:09:21.927097 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:21.927062    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63ed7786-f31d-4538-a5d1-29b328cf5c23/config-reloader/0.log"
Jul 14 16:09:21.950711 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:21.950675    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63ed7786-f31d-4538-a5d1-29b328cf5c23/kube-rbac-proxy-web/0.log"
Jul 14 16:09:21.976371 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:21.976347    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63ed7786-f31d-4538-a5d1-29b328cf5c23/kube-rbac-proxy/0.log"
Jul 14 16:09:22.001998 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.001971    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63ed7786-f31d-4538-a5d1-29b328cf5c23/kube-rbac-proxy-metric/0.log"
Jul 14 16:09:22.032478 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.032452    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63ed7786-f31d-4538-a5d1-29b328cf5c23/prom-label-proxy/0.log"
Jul 14 16:09:22.062524 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.062473    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63ed7786-f31d-4538-a5d1-29b328cf5c23/init-config-reloader/0.log"
Jul 14 16:09:22.147418 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.147388    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-lc7kr_49e14489-568d-455c-97a2-241625d0155a/kube-state-metrics/0.log"
Jul 14 16:09:22.169982 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.169901    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-lc7kr_49e14489-568d-455c-97a2-241625d0155a/kube-rbac-proxy-main/0.log"
Jul 14 16:09:22.200491 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.200464    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-lc7kr_49e14489-568d-455c-97a2-241625d0155a/kube-rbac-proxy-self/0.log"
Jul 14 16:09:22.238626 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.238593    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-59799b7f6c-8zgxl_d05e0184-6720-4dd3-a436-d9033446f7ac/metrics-server/0.log"
Jul 14 16:09:22.274598 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.274571    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-n8zd9_aca900c5-8a1d-4571-b0cb-f0d8ce50f5df/monitoring-plugin/0.log"
Jul 14 16:09:22.397705 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.397675    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7k52l_ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2/node-exporter/0.log"
Jul 14 16:09:22.433186 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.433104    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7k52l_ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2/kube-rbac-proxy/0.log"
Jul 14 16:09:22.464884 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.464854    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7k52l_ac00d550-f9d9-413d-9cc4-3d6afeb9f9b2/init-textfile/0.log"
Jul 14 16:09:22.587805 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.587753    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-w4r2z_95c86f2f-cfd8-4af0-ab06-4812eb0e8886/kube-rbac-proxy-main/0.log"
Jul 14 16:09:22.612845 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.612820    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-w4r2z_95c86f2f-cfd8-4af0-ab06-4812eb0e8886/kube-rbac-proxy-self/0.log"
Jul 14 16:09:22.639781 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.639760    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-w4r2z_95c86f2f-cfd8-4af0-ab06-4812eb0e8886/openshift-state-metrics/0.log"
Jul 14 16:09:22.683174 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.683139    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b88eebc3-6119-4158-8e7a-d6d29f1c0745/prometheus/0.log"
Jul 14 16:09:22.708098 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.708010    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b88eebc3-6119-4158-8e7a-d6d29f1c0745/config-reloader/0.log"
Jul 14 16:09:22.732727 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.732703    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b88eebc3-6119-4158-8e7a-d6d29f1c0745/thanos-sidecar/0.log"
Jul 14 16:09:22.755903 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.755878    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b88eebc3-6119-4158-8e7a-d6d29f1c0745/kube-rbac-proxy-web/0.log"
Jul 14 16:09:22.778931 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.778901    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b88eebc3-6119-4158-8e7a-d6d29f1c0745/kube-rbac-proxy/0.log"
Jul 14 16:09:22.804398 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.804372    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b88eebc3-6119-4158-8e7a-d6d29f1c0745/kube-rbac-proxy-thanos/0.log"
Jul 14 16:09:22.826477 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.826454    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b88eebc3-6119-4158-8e7a-d6d29f1c0745/init-config-reloader/0.log"
Jul 14 16:09:22.858919 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.858891    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-f7bkz_ec57c675-9574-465f-9123-e5c3bda94761/prometheus-operator/0.log"
Jul 14 16:09:22.878652 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.878627    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-f7bkz_ec57c675-9574-465f-9123-e5c3bda94761/kube-rbac-proxy/0.log"
Jul 14 16:09:22.903616 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.903585    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-gdprn_14e8e235-8186-4544-99fa-932691a2df79/prometheus-operator-admission-webhook/0.log"
Jul 14 16:09:22.938709 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.938683    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7599546c9b-5sjmc_817a51df-0e8e-4a0e-af15-981b22389a84/telemeter-client/0.log"
Jul 14 16:09:22.964408 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.964321    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7599546c9b-5sjmc_817a51df-0e8e-4a0e-af15-981b22389a84/reload/0.log"
Jul 14 16:09:22.985886 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:22.985860    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7599546c9b-5sjmc_817a51df-0e8e-4a0e-af15-981b22389a84/kube-rbac-proxy/0.log"
Jul 14 16:09:23.016912 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:23.016884    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69748d48dc-wspq7_998a59ad-a466-47b3-a115-50318de410a3/thanos-query/0.log"
Jul 14 16:09:23.042778 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:23.042748    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69748d48dc-wspq7_998a59ad-a466-47b3-a115-50318de410a3/kube-rbac-proxy-web/0.log"
Jul 14 16:09:23.065418 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:23.065393    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69748d48dc-wspq7_998a59ad-a466-47b3-a115-50318de410a3/kube-rbac-proxy/0.log"
Jul 14 16:09:23.087311 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:23.087282    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69748d48dc-wspq7_998a59ad-a466-47b3-a115-50318de410a3/prom-label-proxy/0.log"
Jul 14 16:09:23.111986 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:23.111960    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69748d48dc-wspq7_998a59ad-a466-47b3-a115-50318de410a3/kube-rbac-proxy-rules/0.log"
Jul 14 16:09:23.134571 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:23.134541    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69748d48dc-wspq7_998a59ad-a466-47b3-a115-50318de410a3/kube-rbac-proxy-metrics/0.log"
Jul 14 16:09:24.630188 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.630154    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"]
Jul 14 16:09:24.630797 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.630776    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5a50bda-98a9-4058-ab8a-fd1d16df1887" containerName="maas-api"
Jul 14 16:09:24.630886 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.630799    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5a50bda-98a9-4058-ab8a-fd1d16df1887" containerName="maas-api"
Jul 14 16:09:24.630886 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.630811    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c94666-30d7-41e3-a345-882cd312cffd" containerName="istio-proxy"
Jul 14 16:09:24.630886 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.630819    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c94666-30d7-41e3-a345-882cd312cffd" containerName="istio-proxy"
Jul 14 16:09:24.630886 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.630833    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0139426-42ac-452e-896b-6250023d5d23" containerName="maas-api"
Jul 14 16:09:24.630886 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.630844    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0139426-42ac-452e-896b-6250023d5d23" containerName="maas-api"
Jul 14 16:09:24.630886 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.630859    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c94666-30d7-41e3-a345-882cd312cffd" containerName="istio-proxy"
Jul 14 16:09:24.630886 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.630867    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c94666-30d7-41e3-a345-882cd312cffd" containerName="istio-proxy"
Jul 14 16:09:24.630886 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.630883    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd5f36b-93a6-456c-8d55-c785a4a98869" containerName="istio-proxy"
Jul 14 16:09:24.631283 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.630892    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd5f36b-93a6-456c-8d55-c785a4a98869" containerName="istio-proxy"
Jul 14 16:09:24.631283 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.630909    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd5f36b-93a6-456c-8d55-c785a4a98869" containerName="istio-proxy"
Jul 14 16:09:24.631283 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.630918    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd5f36b-93a6-456c-8d55-c785a4a98869" containerName="istio-proxy"
Jul 14 16:09:24.631283 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.631009    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd5f36b-93a6-456c-8d55-c785a4a98869" containerName="istio-proxy"
Jul 14 16:09:24.631283 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.631026    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5a50bda-98a9-4058-ab8a-fd1d16df1887" containerName="maas-api"
Jul 14 16:09:24.631283 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.631034    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0139426-42ac-452e-896b-6250023d5d23" containerName="maas-api"
Jul 14 16:09:24.631283 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.631045    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c94666-30d7-41e3-a345-882cd312cffd" containerName="istio-proxy"
Jul 14 16:09:24.631283 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.631053    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c94666-30d7-41e3-a345-882cd312cffd" containerName="istio-proxy"
Jul 14 16:09:24.634240 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.634222    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.637121 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.637099    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zc89m\"/\"kube-root-ca.crt\""
Jul 14 16:09:24.637121 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.637115    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zc89m\"/\"default-dockercfg-khfzj\""
Jul 14 16:09:24.638252 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.638230    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zc89m\"/\"openshift-service-ca.crt\""
Jul 14 16:09:24.643428 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.643405    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"]
Jul 14 16:09:24.651115 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.651082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/efbae6b8-213b-479d-9d1c-e16e546d8d40-podres\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.651245 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.651181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8zpw\" (UniqueName: \"kubernetes.io/projected/efbae6b8-213b-479d-9d1c-e16e546d8d40-kube-api-access-k8zpw\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.651323 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.651259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efbae6b8-213b-479d-9d1c-e16e546d8d40-sys\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.651420 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.651383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/efbae6b8-213b-479d-9d1c-e16e546d8d40-lib-modules\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.651486 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.651461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/efbae6b8-213b-479d-9d1c-e16e546d8d40-proc\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.752448 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.752414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/efbae6b8-213b-479d-9d1c-e16e546d8d40-podres\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.752652 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.752458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8zpw\" (UniqueName: \"kubernetes.io/projected/efbae6b8-213b-479d-9d1c-e16e546d8d40-kube-api-access-k8zpw\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.752652 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.752520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efbae6b8-213b-479d-9d1c-e16e546d8d40-sys\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.752652 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.752564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/efbae6b8-213b-479d-9d1c-e16e546d8d40-lib-modules\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.752652 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.752617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/efbae6b8-213b-479d-9d1c-e16e546d8d40-proc\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.752652 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.752621    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/efbae6b8-213b-479d-9d1c-e16e546d8d40-podres\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.752652 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.752626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efbae6b8-213b-479d-9d1c-e16e546d8d40-sys\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.752861 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.752723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/efbae6b8-213b-479d-9d1c-e16e546d8d40-proc\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.752861 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.752724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/efbae6b8-213b-479d-9d1c-e16e546d8d40-lib-modules\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.762050 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.762033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8zpw\" (UniqueName: \"kubernetes.io/projected/efbae6b8-213b-479d-9d1c-e16e546d8d40-kube-api-access-k8zpw\") pod \"perf-node-gather-daemonset-s6696\" (UID: \"efbae6b8-213b-479d-9d1c-e16e546d8d40\") " pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:24.945252 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:24.945223    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:25.085346 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:25.085315    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"]
Jul 14 16:09:25.086818 ip-10-0-136-210 kubenswrapper[2563]: W0714 16:09:25.086793    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podefbae6b8_213b_479d_9d1c_e16e546d8d40.slice/crio-0e7b231286f6988e72702c7d9341ea72d57be8888b511b16419ce71a9e2471b6 WatchSource:0}: Error finding container 0e7b231286f6988e72702c7d9341ea72d57be8888b511b16419ce71a9e2471b6: Status 404 returned error can't find the container with id 0e7b231286f6988e72702c7d9341ea72d57be8888b511b16419ce71a9e2471b6
Jul 14 16:09:25.563562 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:25.563464    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-75688698d8-jtfr2_d060879f-e91e-40bd-93ea-11c416eaad8a/console/0.log"
Jul 14 16:09:25.837716 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:25.837621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696" event={"ID":"efbae6b8-213b-479d-9d1c-e16e546d8d40","Type":"ContainerStarted","Data":"a897dfc24663139a04630a4c1e2364045ac121b7d7fb5295e30555cf72ce16b6"}
Jul 14 16:09:25.837716 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:25.837672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696" event={"ID":"efbae6b8-213b-479d-9d1c-e16e546d8d40","Type":"ContainerStarted","Data":"0e7b231286f6988e72702c7d9341ea72d57be8888b511b16419ce71a9e2471b6"}
Jul 14 16:09:25.838123 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:25.837718    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:25.856070 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:25.856023    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696" podStartSLOduration=1.856008962 podStartE2EDuration="1.856008962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 16:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 16:09:25.854802747 +0000 UTC m=+2890.196068702" watchObservedRunningTime="2026-07-14 16:09:25.856008962 +0000 UTC m=+2890.197274869"
Jul 14 16:09:27.266215 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:27.266148    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r444t_442ff21f-3879-494d-a460-be0c043a5aa8/dns/0.log"
Jul 14 16:09:27.288102 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:27.288073    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r444t_442ff21f-3879-494d-a460-be0c043a5aa8/kube-rbac-proxy/0.log"
Jul 14 16:09:27.361250 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:27.361219    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9f2sc_b9623835-e777-4e7d-95be-70ce05205902/dns-node-resolver/0.log"
Jul 14 16:09:27.964576 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:27.964530    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7phx7_d2af3c99-9843-496a-823b-f8a97a1dcd9d/node-ca/0.log"
Jul 14 16:09:30.105288 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:30.105235    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-glxgf_f14f969b-df8a-4443-951e-4c9318e2362c/serve-healthcheck-canary/0.log"
Jul 14 16:09:30.751655 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:30.751626    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmlvz_a85dd215-4b19-4732-baf4-d02c27595e38/kube-rbac-proxy/0.log"
Jul 14 16:09:30.773264 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:30.773239    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmlvz_a85dd215-4b19-4732-baf4-d02c27595e38/exporter/0.log"
Jul 14 16:09:30.796615 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:30.796583    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmlvz_a85dd215-4b19-4732-baf4-d02c27595e38/extractor/0.log"
Jul 14 16:09:31.853996 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:31.853970    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zc89m/perf-node-gather-daemonset-s6696"
Jul 14 16:09:32.933544 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:32.933492    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-wg277_0ccbedea-ad39-458c-89af-4fbb1801359a/manager/0.log"
Jul 14 16:09:33.179648 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:33.179616    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wsnwc_1367dbde-4785-4189-a4e5-cb457c93c18b/manager/0.log"
Jul 14 16:09:33.547781 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:33.547753    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66c4b7dd84-7j48t_3e8f4479-2e26-46c8-9b6d-45f72af91d02/manager/0.log"
Jul 14 16:09:33.568544 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:33.568517    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66c4b7dd84-7j48t_3e8f4479-2e26-46c8-9b6d-45f72af91d02/copy-manifests/0.log"
Jul 14 16:09:34.990260 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:34.990223    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d87d6d9c6-ncbx6_d8318403-37b9-4676-95b1-4ea50e076a0d/manager/0.log"
Jul 14 16:09:41.500127 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:41.500097    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bxpqz_5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7/kube-multus-additional-cni-plugins/0.log"
Jul 14 16:09:41.523225 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:41.523198    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bxpqz_5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7/egress-router-binary-copy/0.log"
Jul 14 16:09:41.545678 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:41.545635    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bxpqz_5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7/cni-plugins/0.log"
Jul 14 16:09:41.572343 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:41.572319    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bxpqz_5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7/bond-cni-plugin/0.log"
Jul 14 16:09:41.599330 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:41.599300    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bxpqz_5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7/routeoverride-cni/0.log"
Jul 14 16:09:41.622831 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:41.622805    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bxpqz_5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7/whereabouts-cni-bincopy/0.log"
Jul 14 16:09:41.647624 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:41.647592    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bxpqz_5fecf6a4-135a-41cc-b0d7-cdd720a9d6f7/whereabouts-cni/0.log"
Jul 14 16:09:42.122885 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:42.122852    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l2dj2_93ac7979-c05b-4b5c-8277-d9098a69b8b5/kube-multus/0.log"
Jul 14 16:09:42.231760 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:42.231726    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-frtx9_b42135f6-4895-4ca1-8086-1ae81b8006c8/network-metrics-daemon/0.log"
Jul 14 16:09:42.250568 ip-10-0-136-210 kubenswrapper[2563]: I0714 16:09:42.250538    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-frtx9_b42135f6-4895-4ca1-8086-1ae81b8006c8/kube-rbac-proxy/0.log"