Jul 13 14:50:41.507355 ip-10-0-128-29 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 13 14:50:41.507363 ip-10-0-128-29 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 13 14:50:41.507370 ip-10-0-128-29 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 13 14:50:41.507600 ip-10-0-128-29 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 13 14:50:53.182129 ip-10-0-128-29 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 13 14:50:53.182145 ip-10-0-128-29 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 48d88f68089e4d549253e9c576910bba --
Jul 13 14:52:53.688993 ip-10-0-128-29 systemd[1]: Starting Kubernetes Kubelet...
Jul 13 14:52:54.083267 ip-10-0-128-29 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 14:52:54.083267 ip-10-0-128-29 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 13 14:52:54.083267 ip-10-0-128-29 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 14:52:54.083267 ip-10-0-128-29 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 13 14:52:54.083267 ip-10-0-128-29 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 14:52:54.084162 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.084077    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 13 14:52:54.091308 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091285    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 14:52:54.091308 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091305    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 14:52:54.091308 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091309    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 14:52:54.091308 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091313    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 14:52:54.091308 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091316    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091319    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091322    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091325    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091327    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091330    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091333    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091335    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091338    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091341    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091344    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091346    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091349    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091352    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091354    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091357    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091360    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091362    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091366    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091370    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 14:52:54.091506 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091373    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091376    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091379    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091382    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091386    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091391    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091394    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091403    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091405    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091408    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091410    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091413    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091415    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091418    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091423    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091427    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091430    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091432    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091435    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 14:52:54.092000 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091437    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091440    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091442    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091445    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091447    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091450    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091452    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091455    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091457    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091460    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091462    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091465    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091468    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091470    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091473    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091475    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091477    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091480    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091482    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091485    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 14:52:54.092459 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091487    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091489    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091492    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091494    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091497    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091499    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091501    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091504    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091508    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091511    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091514    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091516    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091519    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091521    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091524    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091526    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091529    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091531    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091533    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 14:52:54.092957 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091536    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091539    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091541    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091543    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091546    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091549    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091551    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091554    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091556    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091559    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091561    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091563    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091566    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091569    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091571    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091573    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.091576    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092279    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092286    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 14:52:54.093415 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092289    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092294    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092298    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092301    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092304    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092307    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092309    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092312    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092314    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092317    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092320    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092322    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092325    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092328    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092330    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092333    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092336    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092338    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092341    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 14:52:54.093885 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092344    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092347    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092350    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092352    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092354    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092357    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092359    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092362    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092365    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092367    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092369    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092372    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092374    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092377    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092382    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092386    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092389    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092391    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092394    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 14:52:54.094400 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092396    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092399    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092402    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092406    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092409    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092412    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092414    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092417    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092419    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092422    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092425    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092427    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092430    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092432    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092435    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092437    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092440    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092442    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092445    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 14:52:54.094897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092447    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092450    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092452    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092455    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092457    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092460    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092462    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092465    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092468    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092471    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092473    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092476    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092478    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092480    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092484    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092486    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092489    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092491    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092493    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092496    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 14:52:54.095408 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092498    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092500    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092503    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092505    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092509    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092512    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092514    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092517    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092520    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092522    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092525    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092527    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092530    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092532    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092535    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092537    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092539    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092541    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092544    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092547    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 14:52:54.095937 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092944    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092950    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092953    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092956    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092959    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092962    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092965    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092968    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092972    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092974    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092977    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092980    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092982    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092985    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092987    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092990    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092993    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092996    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.092998    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093001    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 14:52:54.096430 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093004    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093006    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093009    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093011    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093014    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093016    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093019    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093021    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093024    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093027    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093029    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093032    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093034    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093036    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093039    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093041    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093044    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093046    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093049    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093051    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 14:52:54.096929 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093056    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093059    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093061    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093063    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093066    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093068    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093071    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093074    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093076    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093079    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093081    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093086    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093089    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093093    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093095    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093098    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093100    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093103    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093105    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 14:52:54.097428 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093107    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093110    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093112    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093114    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093117    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093119    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093122    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093124    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093127    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093129    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093132    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093134    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093137    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093140    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093143    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093145    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093147    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093150    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093152    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093155    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 14:52:54.097915 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093157    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093160    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093164    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093168    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093172    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093174    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093177    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093182    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093185    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093188    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093191    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093194    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093196    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093199    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093202    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093204    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093207    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093210    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093213    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093215    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 14:52:54.098405 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093277    2564 flags.go:64] FLAG: --address="0.0.0.0"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093284    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093291    2564 flags.go:64] FLAG: --anonymous-auth="true"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093295    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093299    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093304    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093308    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093312    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093315    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093318    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093321    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093324    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093327    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093330    2564 flags.go:64] FLAG: --cgroup-root=""
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093333    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093336    2564 flags.go:64] FLAG: --client-ca-file=""
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093341    2564 flags.go:64] FLAG: --cloud-provider="external"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093343    2564 flags.go:64] FLAG: --cluster-dns="[]"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093347    2564 flags.go:64] FLAG: --cluster-domain=""
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093350    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093353    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093356    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093359    2564 flags.go:64] FLAG: --container-log-max-files="5"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093363    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 13 14:52:54.098885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093365    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093369    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093371    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093374    2564 flags.go:64] FLAG: --contention-profiling="false"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093377    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093380    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093383    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093386    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093389    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093392    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093395    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093398    2564 flags.go:64] FLAG: --enable-load-reader="false"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093401    2564 flags.go:64] FLAG: --enable-server="true"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093405    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093409    2564 flags.go:64] FLAG: --event-burst="100"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093412    2564 flags.go:64] FLAG: --event-qps="50"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093415    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093418    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093421    2564 flags.go:64] FLAG: --eviction-hard=""
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093425    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093428    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093430    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093434    2564 flags.go:64] FLAG: --eviction-soft=""
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093436    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093440    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 13 14:52:54.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093443    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093446    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093449    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093451    2564 flags.go:64] FLAG: --fail-swap-on="true"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093454    2564 flags.go:64] FLAG: --feature-gates=""
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093460    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093463    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093466    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093468    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093471    2564 flags.go:64] FLAG: --healthz-port="10248"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093474    2564 flags.go:64] FLAG: --help="false"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093476    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093479    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093482    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093485    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093488    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093491    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093494    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093497    2564 flags.go:64] FLAG: --image-service-endpoint=""
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093499    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093503    2564 flags.go:64] FLAG: --kube-api-burst="100"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093506    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093509    2564 flags.go:64] FLAG: --kube-api-qps="50"
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093512    2564 flags.go:64] FLAG: --kube-reserved=""
Jul 13 14:52:54.100202 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093514    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093517    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093520    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093523    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093526    2564 flags.go:64] FLAG: --lock-file=""
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093528    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093531    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093534    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093539    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093543    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093546    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093549    2564 flags.go:64] FLAG: --logging-format="text"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093551    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093554    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093557    2564 flags.go:64] FLAG: --manifest-url=""
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093560    2564 flags.go:64] FLAG: --manifest-url-header=""
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093564    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093567    2564 flags.go:64] FLAG: --max-open-files="1000000"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093570    2564 flags.go:64] FLAG: --max-pods="110"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093573    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093576    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093579    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093582    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093584    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093587    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 13 14:52:54.100829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093590    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093597    2564 flags.go:64] FLAG: --node-status-max-images="50"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093600    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093604    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093607    2564 flags.go:64] FLAG: --pod-cidr=""
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093610    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093615    2564 flags.go:64] FLAG: --pod-manifest-path=""
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093617    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093620    2564 flags.go:64] FLAG: --pods-per-core="0"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093623    2564 flags.go:64] FLAG: --port="10250"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093626    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093628    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-001760880f675ec59"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093631    2564 flags.go:64] FLAG: --qos-reserved=""
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093634    2564 flags.go:64] FLAG: --read-only-port="10255"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093637    2564 flags.go:64] FLAG: --register-node="true"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093639    2564 flags.go:64] FLAG: --register-with-taints=""
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093642    2564 flags.go:64] FLAG: --registry-burst="10"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093645    2564 flags.go:64] FLAG: --registry-qps="5"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093648    2564 flags.go:64] FLAG: --reserved-cpus=""
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093651    2564 flags.go:64] FLAG: --reserved-memory=""
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093654    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093657    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093660    2564 flags.go:64] FLAG: --rotate-certificates="false"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093662    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093665    2564 flags.go:64] FLAG: --runonce="false"
Jul 13 14:52:54.101439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093667    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093670    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093673    2564 flags.go:64] FLAG: --seccomp-default="false"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093676    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093678    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093681    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093684    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093687    2564 flags.go:64] FLAG: --storage-driver-password="root"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093689    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093706    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093710    2564 flags.go:64] FLAG: --storage-driver-user="root"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093713    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093716    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093719    2564 flags.go:64] FLAG: --system-cgroups=""
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093721    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093726    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093729    2564 flags.go:64] FLAG: --tls-cert-file=""
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093732    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093735    2564 flags.go:64] FLAG: --tls-min-version=""
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093738    2564 flags.go:64] FLAG: --tls-private-key-file=""
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093741    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093744    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093746    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093749    2564 flags.go:64] FLAG: --v="2"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093753    2564 flags.go:64] FLAG: --version="false"
Jul 13 14:52:54.102080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093757    2564 flags.go:64] FLAG: --vmodule=""
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093761    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.093764    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093879    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093882    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093885    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093888    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093891    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093893    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093896    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093899    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093901    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093905    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093908    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093910    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093913    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093915    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093919    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093921    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093923    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 14:52:54.102738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093926    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093928    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093931    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093934    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093936    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093939    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093941    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093944    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093947    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093949    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093952    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093954    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093956    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093959    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093962    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093964    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093966    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093969    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093971    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093974    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 14:52:54.103234 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093976    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093980    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093983    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093986    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093990    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093993    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.093997    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094000    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094003    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094007    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094010    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094013    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094016    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094018    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094021    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094023    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094026    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094028    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094030    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 14:52:54.103738 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094033    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094035    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094038    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094040    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094043    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094046    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094049    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094053    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094055    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094058    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094060    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094063    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094065    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094067    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094069    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094072    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094074    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094078    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094080    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094083    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 14:52:54.104194 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094085    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094088    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094091    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094093    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094096    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094099    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094101    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094103    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094106    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094108    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094110    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094113    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094115    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094117    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094120    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094122    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094124    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094127    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094129    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 14:52:54.104675 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094132    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 14:52:54.105138 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094135    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 14:52:54.105138 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094137    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 14:52:54.105138 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.094139    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 14:52:54.105138 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.094727    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 14:52:54.105138 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.102015    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 13 14:52:54.105138 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.102038    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 13 14:52:54.105138 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.102054    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 13 14:52:54.105138 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.102059    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 13 14:52:54.105138 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102129    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 14:52:54.105138 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102136    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 14:52:54.105138 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102140    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 14:52:54.105138 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102143    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 14:52:54.105138 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102146    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 14:52:54.105138 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102149    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102151    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102154    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102157    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102159    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102162    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102165    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102167    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102170    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102173    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102183    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102186    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102189    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102193    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102198    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102202    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102207    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102211    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102214    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102216    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 14:52:54.105490 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102219    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102221    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102224    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102226    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102236    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102239    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102242    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102244    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102247    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102250    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102252    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102254    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102257    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102259    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102262    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102264    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102266    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102269    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102271    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102275    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 14:52:54.105985 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102279    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102284    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102288    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102292    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102296    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102300    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102303    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102305    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102308    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102311    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102314    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102316    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102319    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102321    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102323    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102327    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102329    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102339    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102342    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102344    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 14:52:54.106479 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102347    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102350    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102352    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102355    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102360    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102364    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102368    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102372    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102375    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102378    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102380    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102382    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102385    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102388    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102391    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102394    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102397    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102399    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102402    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102405    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 14:52:54.106983 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102407    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102410    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102412    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102414    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102417    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102419    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102421    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102424    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102426    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102429    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102438    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102443    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102447    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102451    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.102459    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 14:52:54.107464 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102625    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102630    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102633    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102635    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102638    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102641    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102643    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102646    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102648    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102650    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102653    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102655    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102657    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102660    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102662    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102664    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102666    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102669    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102671    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102673    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 14:52:54.107857 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102676    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102678    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102680    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102683    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102685    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102689    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102711    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102714    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102724    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102726    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102729    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102731    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102734    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102736    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102738    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102741    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102745    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102748    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102751    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 14:52:54.108342 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102754    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102757    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102760    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102762    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102765    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102768    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102771    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102776    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102780    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102784    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102788    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102792    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102795    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102797    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102799    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102803    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102806    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102810    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102813    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 14:52:54.108837 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102816    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102819    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102822    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102830    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102833    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102836    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102839    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102841    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102843    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102846    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102848    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102850    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102853    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102858    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102862    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102866    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102870    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102874    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102876    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102879    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 14:52:54.109309 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102881    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102883    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102886    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102888    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102890    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102892    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102895    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102897    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102900    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102902    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102904    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102907    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102909    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102912    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102915    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102917    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102925    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102928    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102930    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102933    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 14:52:54.109839 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:54.102935    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 14:52:54.110643 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.102943    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 14:52:54.110643 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.103084    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 13 14:52:54.110643 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.105108    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 13 14:52:54.110643 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.106529    2564 server.go:1019] "Starting client certificate rotation"
Jul 13 14:52:54.110643 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.106632    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 14:52:54.110643 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.106665    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 14:52:54.110643 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.109128    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 14:52:54.114915 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.114893    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 14:52:54.126470 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.126441    2564 log.go:25] "Validated CRI v1 runtime API"
Jul 13 14:52:54.128738 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.128723    2564 log.go:25] "Validated CRI v1 image API"
Jul 13 14:52:54.129543 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.129530    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 13 14:52:54.132607 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.132589    2564 fs.go:135] Filesystem UUIDs: map[02c57447-e9a1-4d42-b643-eb51c0d89da0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e290a024-3da4-4871-940e-886d5a861b2c:/dev/nvme0n1p3]
Jul 13 14:52:54.132676 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.132606    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 13 14:52:54.136280 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.136263    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 14:52:54.137964 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.137857    2564 manager.go:216] Machine: {Timestamp:2026-07-13 14:52:54.13634307 +0000 UTC m=+0.333496008 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097859 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25d24c61266f8c4846c3d61ac379a3 SystemUUID:ec25d24c-6126-6f8c-4846-c3d61ac379a3 BootID:48d88f68-089e-4d54-9253-e9c576910bba Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:44:dd:03:2c:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:44:dd:03:2c:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:37:6a:7b:30:0f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 13 14:52:54.137964 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.137961    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 13 14:52:54.138072 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.138039    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 13 14:52:54.140845 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.140824    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 13 14:52:54.141002 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.140849    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-29.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 13 14:52:54.141573 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.141564    2564 topology_manager.go:138] "Creating topology manager with none policy"
Jul 13 14:52:54.141610 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.141575    2564 container_manager_linux.go:309] "Creating device plugin manager"
Jul 13 14:52:54.141610 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.141583    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 14:52:54.142149 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.142139    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 14:52:54.142995 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.142985    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 13 14:52:54.145770 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.145759    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 14:52:54.145889 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.145880    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 13 14:52:54.145921 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.145912    2564 kubelet.go:486] "Attempting to sync node with API server"
Jul 13 14:52:54.145948 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.145921    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 13 14:52:54.145948 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.145935    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 13 14:52:54.145948 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.145943    2564 kubelet.go:398] "Adding apiserver pod source"
Jul 13 14:52:54.146030 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.145958    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 13 14:52:54.146447 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.146434    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 14:52:54.146494 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.146454    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 14:52:54.148976 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.148960    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 13 14:52:54.150356 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.150340    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 13 14:52:54.150433 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.150379    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 13 14:52:54.151453 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.151440    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 13 14:52:54.151531 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.151459    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 13 14:52:54.151531 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.151468    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 13 14:52:54.151531 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.151477    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 13 14:52:54.151531 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.151489    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 13 14:52:54.151531 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.151498    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 13 14:52:54.151531 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.151509    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 13 14:52:54.151531 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.151517    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 13 14:52:54.151531 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.151527    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 13 14:52:54.151531 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.151535    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 13 14:52:54.151834 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.151552    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 13 14:52:54.151834 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.151560    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 13 14:52:54.152312 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.152301    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 13 14:52:54.152360 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.152317    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 13 14:52:54.155475 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.155459    2564 server.go:1268] "Started kubelet"
Jul 13 14:52:54.155558 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.155528    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 13 14:52:54.159819 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.159453    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 13 14:52:54.159819 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.159535    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 13 14:52:54.159819 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.159786    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6hmnz"
Jul 13 14:52:54.159986 ip-10-0-128-29 systemd[1]: Started Kubernetes Kubelet.
Jul 13 14:52:54.161912 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.161094    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 13 14:52:54.162621 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.162597    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-29.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 13 14:52:54.162727 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.162605    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 13 14:52:54.162727 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.162674    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 13 14:52:54.162945 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.162927    2564 server.go:310] "Adding debug handlers to kubelet server"
Jul 13 14:52:54.166275 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.166255    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 13 14:52:54.166750 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.166735    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6hmnz"
Jul 13 14:52:54.168785 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.168767    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 13 14:52:54.168865 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.168774    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 13 14:52:54.168929 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.168884    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Jul 13 14:52:54.169653 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.169629    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 13 14:52:54.169653 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.169653    2564 factory.go:55] Registering systemd factory
Jul 13 14:52:54.169824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.169662    2564 factory.go:223] Registration of the systemd container factory successfully
Jul 13 14:52:54.169824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.169631    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 13 14:52:54.169885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.169854    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 13 14:52:54.169885 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.169867    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 13 14:52:54.170117 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.170101    2564 factory.go:153] Registering CRI-O factory
Jul 13 14:52:54.170117 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.170116    2564 factory.go:223] Registration of the crio container factory successfully
Jul 13 14:52:54.170117 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.170134    2564 factory.go:103] Registering Raw factory
Jul 13 14:52:54.170322 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.170147    2564 manager.go:1196] Started watching for new ooms in manager
Jul 13 14:52:54.170758 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.170735    2564 reconstruct.go:97] "Volume reconstruction finished"
Jul 13 14:52:54.170758 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.170752    2564 manager.go:349] Starting recovery of all containers
Jul 13 14:52:54.171269 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.170754    2564 reconciler.go:29] "Reconciler: start to sync state"
Jul 13 14:52:54.172886 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.172860    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 13 14:52:54.174000 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.173972    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 13 14:52:54.174427 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.173134    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-29.ec2.internal.18c1e10d2f09e3c7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-29.ec2.internal,UID:ip-10-0-128-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-29.ec2.internal,},FirstTimestamp:2026-07-13 14:52:54.155428807 +0000 UTC m=+0.352581746,LastTimestamp:2026-07-13 14:52:54.155428807 +0000 UTC m=+0.352581746,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-29.ec2.internal,}"
Jul 13 14:52:54.180732 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.180581    2564 manager.go:354] Recovery completed
Jul 13 14:52:54.185339 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.185327    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 14:52:54.187112 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.187096    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 14:52:54.187172 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.187129    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 14:52:54.187172 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.187138    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Jul 13 14:52:54.187632 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.187621    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 13 14:52:54.187632 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.187632    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 13 14:52:54.187751 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.187647    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 14:52:54.187751 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.187678    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 14:52:54.188439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.188414    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 14:52:54.188508 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.188452    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 14:52:54.188508 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.188466    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Jul 13 14:52:54.188829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.188817    2564 policy_none.go:49] "None policy: Start"
Jul 13 14:52:54.188886 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.188833    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 13 14:52:54.188886 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.188844    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 13 14:52:54.189870 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.189856    2564 policy_none.go:47] "Start"
Jul 13 14:52:54.229785 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.229769    2564 manager.go:337] "Starting Device Plugin manager"
Jul 13 14:52:54.229881 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.229841    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 13 14:52:54.229881 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.229852    2564 server.go:87] "Starting device plugin registration server"
Jul 13 14:52:54.230099 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.230088    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 13 14:52:54.230152 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.230100    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 13 14:52:54.230221 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.230184    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 13 14:52:54.230277 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.230264    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 13 14:52:54.230320 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.230276    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 13 14:52:54.230846 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.230822    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 13 14:52:54.230929 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.230883    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-29.ec2.internal\" not found"
Jul 13 14:52:54.308932 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.308895    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 13 14:52:54.310280 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.310261    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 13 14:52:54.310373 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.310298    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 13 14:52:54.310373 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.310320    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 13 14:52:54.310373 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.310355    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 13 14:52:54.312888 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.312869    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 14:52:54.330939 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.330918    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 14:52:54.331797 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.331781    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 14:52:54.331919 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.331812    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 14:52:54.331919 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.331823    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Jul 13 14:52:54.331919 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.331848    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.337912 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.337867    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.337912 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.337887    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-29.ec2.internal\": node \"ip-10-0-128-29.ec2.internal\" not found"
Jul 13 14:52:54.352105 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.352086    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Jul 13 14:52:54.411402 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.411381    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"]
Jul 13 14:52:54.411470 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.411434    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 14:52:54.412795 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.412781    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 14:52:54.412864 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.412807    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 14:52:54.412864 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.412817    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Jul 13 14:52:54.414142 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.414130    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 14:52:54.414273 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.414258    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.414318 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.414288    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 14:52:54.414889 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.414868    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 14:52:54.414889 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.414881    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 14:52:54.415007 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.414904    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 14:52:54.415007 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.414904    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 14:52:54.415007 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.414936    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Jul 13 14:52:54.415007 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.414918    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Jul 13 14:52:54.416361 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.416346    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.416406 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.416373    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 14:52:54.417024 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.417008    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 14:52:54.417088 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.417039    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 14:52:54.417088 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.417056    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Jul 13 14:52:54.435270 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.435245    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-29.ec2.internal\" not found" node="ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.439115 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.439099    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-29.ec2.internal\" not found" node="ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.452524 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.452505    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Jul 13 14:52:54.473136 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.473113    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fe4ee3c51c80837e4836133e719eda0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-29.ec2.internal\" (UID: \"9fe4ee3c51c80837e4836133e719eda0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.473210 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.473141    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4560c566f29ffa1717b75b3bb741078b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"4560c566f29ffa1717b75b3bb741078b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.473210 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.473158    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4560c566f29ffa1717b75b3bb741078b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"4560c566f29ffa1717b75b3bb741078b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.552605 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.552577    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Jul 13 14:52:54.573929 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.573902    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fe4ee3c51c80837e4836133e719eda0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-29.ec2.internal\" (UID: \"9fe4ee3c51c80837e4836133e719eda0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.574025 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.573936    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4560c566f29ffa1717b75b3bb741078b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"4560c566f29ffa1717b75b3bb741078b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.574025 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.573962    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4560c566f29ffa1717b75b3bb741078b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"4560c566f29ffa1717b75b3bb741078b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.574025 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.574013    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fe4ee3c51c80837e4836133e719eda0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-29.ec2.internal\" (UID: \"9fe4ee3c51c80837e4836133e719eda0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.574025 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.574015    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4560c566f29ffa1717b75b3bb741078b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"4560c566f29ffa1717b75b3bb741078b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.574164 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.574018    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4560c566f29ffa1717b75b3bb741078b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"4560c566f29ffa1717b75b3bb741078b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.653338 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.653312    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Jul 13 14:52:54.737920 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.737896    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.742339 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:54.742262    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:54.754220 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.754199    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Jul 13 14:52:54.854864 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.854826    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Jul 13 14:52:54.955391 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:54.955323    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Jul 13 14:52:55.055881 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:55.055859    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Jul 13 14:52:55.106123 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:55.106095    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 13 14:52:55.106574 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:55.106226    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 14:52:55.156401 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:55.156374    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Jul 13 14:52:55.171600 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:55.171563    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 14:47:54 +0000 UTC" deadline="2028-02-22 03:36:14.405660422 +0000 UTC"
Jul 13 14:52:55.171600 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:55.171594    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14124h43m19.234069528s"
Jul 13 14:52:55.207407 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:55.207350    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 14:52:55.257140 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:55.257112    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Jul 13 14:52:55.315936 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:55.315903    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4560c566f29ffa1717b75b3bb741078b.slice/crio-8f6990f82bec2f8e208b2fe95c26990c180d0dcfd378b67a2bb5619470d2392e WatchSource:0}: Error finding container 8f6990f82bec2f8e208b2fe95c26990c180d0dcfd378b67a2bb5619470d2392e: Status 404 returned error can't find the container with id 8f6990f82bec2f8e208b2fe95c26990c180d0dcfd378b67a2bb5619470d2392e
Jul 13 14:52:55.319608 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:55.319594    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 14:52:55.357508 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:55.357481    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Jul 13 14:52:55.442285 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:55.442255    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 14:52:55.453058 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:55.453034    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 14:52:55.469767 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:55.469715    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:55.479504 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:55.479482    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 14:52:55.480469 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:55.480457    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Jul 13 14:52:55.490956 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:55.490937    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 14:52:56.147152 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.147118    2564 apiserver.go:52] "Watching apiserver"
Jul 13 14:52:56.155465 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.155438    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 13 14:52:56.155892 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.155867    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-5xcbw","kube-system/global-pull-secret-syncer-jgpsr","kube-system/konnectivity-agent-tjrpn","kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk","openshift-cluster-node-tuning-operator/tuned-fprbx","openshift-dns/node-resolver-ptz2c","openshift-multus/multus-additional-cni-plugins-wgw48","openshift-ovn-kubernetes/ovnkube-node-tfgpx","openshift-image-registry/node-ca-k9gzt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal","openshift-multus/multus-rk6vg","openshift-multus/network-metrics-daemon-9pghm","openshift-network-diagnostics/network-check-target-hl7zq"]
Jul 13 14:52:56.157561 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.157542    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k9gzt"
Jul 13 14:52:56.158962 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.158940    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:52:56.159063 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.159029    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:52:56.159920 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.159898    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sb2kk\""
Jul 13 14:52:56.160019 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.159992    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 13 14:52:56.160101 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.160082    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 13 14:52:56.160297 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.160279    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 13 14:52:56.160479 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.160458    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tjrpn"
Jul 13 14:52:56.161588 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.161571    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:52:56.161679 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.161633    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:52:56.162835 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.162813    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 13 14:52:56.163057 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.163034    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tc4l2\""
Jul 13 14:52:56.163144 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.163090    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 13 14:52:56.163252 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.163238    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.163935 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.163917    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.165389 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.165371    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.166660 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.166642    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tr8cz\""
Jul 13 14:52:56.166854 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.166837    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 13 14:52:56.166975 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.166963    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 13 14:52:56.167106 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.167094    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 13 14:52:56.167247 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.167236    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 13 14:52:56.167363 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.167345    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ptz2c"
Jul 13 14:52:56.167415 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.167401    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 13 14:52:56.167542 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.167531    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jwlfp\""
Jul 13 14:52:56.168622 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.167984    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 13 14:52:56.168622 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.167990    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 13 14:52:56.168622 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.168298    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:52:56.168622 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.168308    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 13 14:52:56.168622 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.168349    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:52:56.168622 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.168362    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 13 14:52:56.168974 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.168889    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 13 14:52:56.169028 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.168997    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 13 14:52:56.169078 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.169045    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 13 14:52:56.169078 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.168998    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4cfp4\""
Jul 13 14:52:56.169473 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.169455    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5xcbw"
Jul 13 14:52:56.169976 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.169949    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5zfbr\""
Jul 13 14:52:56.170060 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.169958    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 13 14:52:56.170060 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.170006    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 13 14:52:56.171278 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.171260    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.171908 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.171869    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 13 14:52:56.172255 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.172236    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sxjpg\""
Jul 13 14:52:56.172601 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.172364    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 13 14:52:56.173055 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.173020    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 13 14:52:56.173147 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.173127    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.174094 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.174075    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 13 14:52:56.176258 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.174510    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 13 14:52:56.176258 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.174649    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 13 14:52:56.176258 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.175219    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 13 14:52:56.176258 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.175393    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 13 14:52:56.176258 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.175784    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hj5cd\""
Jul 13 14:52:56.176258 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.175897    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 13 14:52:56.176518 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.176399    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gq9qr\""
Jul 13 14:52:56.181027 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.180995    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-run-ovn-kubernetes\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.181115 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181030    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-etc-selinux\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.181115 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181054    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84f22329-522f-4502-a166-d0f7139138fc-hosts-file\") pod \"node-resolver-ptz2c\" (UID: \"84f22329-522f-4502-a166-d0f7139138fc\") " pod="openshift-dns/node-resolver-ptz2c"
Jul 13 14:52:56.181115 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181091    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:52:56.181269 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181144    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nxmz\" (UniqueName: \"kubernetes.io/projected/7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5-kube-api-access-8nxmz\") pod \"node-ca-k9gzt\" (UID: \"7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5\") " pod="openshift-image-registry/node-ca-k9gzt"
Jul 13 14:52:56.181269 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181169    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-systemd-units\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.181269 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-kubernetes\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.181269 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181226    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-sysctl-d\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.181269 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181247    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-run-openvswitch\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.181269 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181269    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-env-overrides\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.181467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181282    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 14:52:56.181467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181285    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6542e3b-cdda-4bd2-9419-667e66525978-system-cni-dir\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.181467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181326    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bd78a31-4667-420c-9ed6-52df573dbaf5-agent-certs\") pod \"konnectivity-agent-tjrpn\" (UID: \"4bd78a31-4667-420c-9ed6-52df573dbaf5\") " pod="kube-system/konnectivity-agent-tjrpn"
Jul 13 14:52:56.181467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181346    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-host\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.181467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181364    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-kubelet\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.181467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181386    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-cni-netd\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.181467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181411    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6542e3b-cdda-4bd2-9419-667e66525978-cnibin\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.181467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84f22329-522f-4502-a166-d0f7139138fc-tmp-dir\") pod \"node-resolver-ptz2c\" (UID: \"84f22329-522f-4502-a166-d0f7139138fc\") " pod="openshift-dns/node-resolver-ptz2c"
Jul 13 14:52:56.181768 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181473    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdef54cb-b0a3-4783-9655-5296a4f95022-kubelet-config\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:52:56.181768 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6542e3b-cdda-4bd2-9419-667e66525978-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.181768 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181509    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lbgj\" (UniqueName: \"kubernetes.io/projected/49b6a1b2-b8e5-4228-b018-25bc21f3add6-kube-api-access-2lbgj\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.181768 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181524    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fnls\" (UniqueName: \"kubernetes.io/projected/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-kube-api-access-6fnls\") pod \"network-metrics-daemon-9pghm\" (UID: \"93029547-ab84-4b4f-ac0f-e1b86f5ca5de\") " pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:52:56.181768 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181545    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.181768 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181559    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-ovnkube-script-lib\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.181768 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181572    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdef54cb-b0a3-4783-9655-5296a4f95022-dbus\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:52:56.181768 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181584    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-run\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.181768 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181601    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-sys\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.181768 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181665    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6395f961-088a-4295-b37d-8e6660b93365-host-slash\") pod \"iptables-alerter-5xcbw\" (UID: \"6395f961-088a-4295-b37d-8e6660b93365\") " pod="openshift-network-operator/iptables-alerter-5xcbw"
Jul 13 14:52:56.181768 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-run-netns\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.181768 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-device-dir\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181774    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j57h2\" (UniqueName: \"kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2\") pod \"network-check-target-hl7zq\" (UID: \"af9196c9-6c2a-466c-bea6-fd690110ef71\") " pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181791    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5-serviceca\") pod \"node-ca-k9gzt\" (UID: \"7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5\") " pod="openshift-image-registry/node-ca-k9gzt"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181809    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-var-lib-kubelet\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-run-systemd\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181844    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181876    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6395f961-088a-4295-b37d-8e6660b93365-iptables-alerter-script\") pod \"iptables-alerter-5xcbw\" (UID: \"6395f961-088a-4295-b37d-8e6660b93365\") " pod="openshift-network-operator/iptables-alerter-5xcbw"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181894    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-slash\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181912    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-node-log\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181932    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-ovnkube-config\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181953    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pknt5\" (UniqueName: \"kubernetes.io/projected/84f22329-522f-4502-a166-d0f7139138fc-kube-api-access-pknt5\") pod \"node-resolver-ptz2c\" (UID: \"84f22329-522f-4502-a166-d0f7139138fc\") " pod="openshift-dns/node-resolver-ptz2c"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181975    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-sysconfig\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.181995    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbfx4\" (UniqueName: \"kubernetes.io/projected/f6542e3b-cdda-4bd2-9419-667e66525978-kube-api-access-rbfx4\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182019    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bd78a31-4667-420c-9ed6-52df573dbaf5-konnectivity-ca\") pod \"konnectivity-agent-tjrpn\" (UID: \"4bd78a31-4667-420c-9ed6-52df573dbaf5\") " pod="kube-system/konnectivity-agent-tjrpn"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-sys-fs\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182058    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-var-lib-openvswitch\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.182178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182094    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-run-ovn\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182123    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-cni-bin\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182144    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6542e3b-cdda-4bd2-9419-667e66525978-os-release\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182167    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6542e3b-cdda-4bd2-9419-667e66525978-cni-binary-copy\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182187    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-modprobe-d\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182206    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs\") pod \"network-metrics-daemon-9pghm\" (UID: \"93029547-ab84-4b4f-ac0f-e1b86f5ca5de\") " pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182227    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/087b94d6-aea4-4f87-8c4a-cd199129891d-tmp\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182253    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tb7g\" (UniqueName: \"kubernetes.io/projected/087b94d6-aea4-4f87-8c4a-cd199129891d-kube-api-access-8tb7g\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182281    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-log-socket\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182302    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-ovn-node-metrics-cert\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182322    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6542e3b-cdda-4bd2-9419-667e66525978-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182341    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td4xb\" (UniqueName: \"kubernetes.io/projected/6395f961-088a-4295-b37d-8e6660b93365-kube-api-access-td4xb\") pod \"iptables-alerter-5xcbw\" (UID: \"6395f961-088a-4295-b37d-8e6660b93365\") " pod="openshift-network-operator/iptables-alerter-5xcbw"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182367    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5-host\") pod \"node-ca-k9gzt\" (UID: \"7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5\") " pod="openshift-image-registry/node-ca-k9gzt"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182385    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6542e3b-cdda-4bd2-9419-667e66525978-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182418    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-socket-dir\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182435    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-sysctl-conf\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.182734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182462    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-lib-modules\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.183338 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-tuned\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.183338 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182514    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-etc-openvswitch\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.183338 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182535    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mqj5\" (UniqueName: \"kubernetes.io/projected/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-kube-api-access-5mqj5\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.183338 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182556    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-registration-dir\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.183338 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.182572    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-systemd\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.211045 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.211019    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xrvw"
Jul 13 14:52:56.217340 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.217315    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xrvw"
Jul 13 14:52:56.270789 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.270765    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 13 14:52:56.282789 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.282761    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57h2\" (UniqueName: \"kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2\") pod \"network-check-target-hl7zq\" (UID: \"af9196c9-6c2a-466c-bea6-fd690110ef71\") " pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:52:56.282933 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.282797    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5-serviceca\") pod \"node-ca-k9gzt\" (UID: \"7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5\") " pod="openshift-image-registry/node-ca-k9gzt"
Jul 13 14:52:56.282933 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.282823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-var-lib-kubelet\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.282933 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.282877    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-run-systemd\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.282933 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.282912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.283130 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.282940    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6395f961-088a-4295-b37d-8e6660b93365-iptables-alerter-script\") pod \"iptables-alerter-5xcbw\" (UID: \"6395f961-088a-4295-b37d-8e6660b93365\") " pod="openshift-network-operator/iptables-alerter-5xcbw"
Jul 13 14:52:56.283130 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.282978    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-run-systemd\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.283130 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.282992    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.283130 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283017    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-slash\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.283130 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283045    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-node-log\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.283130 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-ovnkube-config\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.283130 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283094    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-cnibin\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.283130 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283117    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-multus-conf-dir\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.283357 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283141    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pknt5\" (UniqueName: \"kubernetes.io/projected/84f22329-522f-4502-a166-d0f7139138fc-kube-api-access-pknt5\") pod \"node-resolver-ptz2c\" (UID: \"84f22329-522f-4502-a166-d0f7139138fc\") " pod="openshift-dns/node-resolver-ptz2c"
Jul 13 14:52:56.283357 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283166    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-sysconfig\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.283357 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283190    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbfx4\" (UniqueName: \"kubernetes.io/projected/f6542e3b-cdda-4bd2-9419-667e66525978-kube-api-access-rbfx4\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.283357 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283216    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-system-cni-dir\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.283357 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283225    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-var-lib-kubelet\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.283357 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283238    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-multus-daemon-config\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.283357 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bd78a31-4667-420c-9ed6-52df573dbaf5-konnectivity-ca\") pod \"konnectivity-agent-tjrpn\" (UID: \"4bd78a31-4667-420c-9ed6-52df573dbaf5\") " pod="kube-system/konnectivity-agent-tjrpn"
Jul 13 14:52:56.283357 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283280    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-slash\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.283357 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-sys-fs\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.283640 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283364    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-var-lib-openvswitch\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.283640 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283392    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-run-ovn\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.283640 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283417    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-cni-bin\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.283640 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6542e3b-cdda-4bd2-9419-667e66525978-os-release\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.283640 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283465    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6542e3b-cdda-4bd2-9419-667e66525978-cni-binary-copy\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.283640 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283490    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-modprobe-d\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.283640 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283504    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5-serviceca\") pod \"node-ca-k9gzt\" (UID: \"7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5\") " pod="openshift-image-registry/node-ca-k9gzt"
Jul 13 14:52:56.283640 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283516    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs\") pod \"network-metrics-daemon-9pghm\" (UID: \"93029547-ab84-4b4f-ac0f-e1b86f5ca5de\") " pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:52:56.283640 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283541    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/087b94d6-aea4-4f87-8c4a-cd199129891d-tmp\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.283640 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283578    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tb7g\" (UniqueName: \"kubernetes.io/projected/087b94d6-aea4-4f87-8c4a-cd199129891d-kube-api-access-8tb7g\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.283640 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283602    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-log-socket\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.283640 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283626    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-ovn-node-metrics-cert\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.284151 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283644    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-sysconfig\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.284151 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283651    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6542e3b-cdda-4bd2-9419-667e66525978-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.284151 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283645    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-node-log\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.284151 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283857    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-log-socket\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.284151 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283602    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6395f961-088a-4295-b37d-8e6660b93365-iptables-alerter-script\") pod \"iptables-alerter-5xcbw\" (UID: \"6395f961-088a-4295-b37d-8e6660b93365\") " pod="openshift-network-operator/iptables-alerter-5xcbw"
Jul 13 14:52:56.284151 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6542e3b-cdda-4bd2-9419-667e66525978-os-release\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.284151 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284007    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td4xb\" (UniqueName: \"kubernetes.io/projected/6395f961-088a-4295-b37d-8e6660b93365-kube-api-access-td4xb\") pod \"iptables-alerter-5xcbw\" (UID: \"6395f961-088a-4295-b37d-8e6660b93365\") " pod="openshift-network-operator/iptables-alerter-5xcbw"
Jul 13 14:52:56.284151 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284072    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-run-ovn\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.284151 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283943    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-sys-fs\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.284151 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284104    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5-host\") pod \"node-ca-k9gzt\" (UID: \"7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5\") " pod="openshift-image-registry/node-ca-k9gzt"
Jul 13 14:52:56.284151 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-cni-bin\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.284642 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.283602    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-var-lib-openvswitch\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.284642 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284152    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-run-k8s-cni-cncf-io\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.284642 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284257    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-modprobe-d\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.284642 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6542e3b-cdda-4bd2-9419-667e66525978-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.284642 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284297    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 13 14:52:56.284642 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284347    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-ovnkube-config\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.284642 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284336    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-multus-socket-dir-parent\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.284642 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284543    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bd78a31-4667-420c-9ed6-52df573dbaf5-konnectivity-ca\") pod \"konnectivity-agent-tjrpn\" (UID: \"4bd78a31-4667-420c-9ed6-52df573dbaf5\") " pod="kube-system/konnectivity-agent-tjrpn"
Jul 13 14:52:56.284642 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.284558    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 14:52:56.284642 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.284628    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs podName:93029547-ab84-4b4f-ac0f-e1b86f5ca5de nodeName:}" failed. No retries permitted until 2026-07-13 14:52:56.784606682 +0000 UTC m=+2.981759627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs") pod "network-metrics-daemon-9pghm" (UID: "93029547-ab84-4b4f-ac0f-e1b86f5ca5de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 14:52:56.284642 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284634    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6542e3b-cdda-4bd2-9419-667e66525978-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284686    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-socket-dir\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284744    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5-host\") pod \"node-ca-k9gzt\" (UID: \"7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5\") " pod="openshift-image-registry/node-ca-k9gzt"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-sysctl-conf\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284888    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6542e3b-cdda-4bd2-9419-667e66525978-cni-binary-copy\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-lib-modules\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284900    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-socket-dir\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284949    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-tuned\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284991    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-etc-openvswitch\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.284996    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-sysctl-conf\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285003    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-lib-modules\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mqj5\" (UniqueName: \"kubernetes.io/projected/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-kube-api-access-5mqj5\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285041    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-registration-dir\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285051    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-etc-openvswitch\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285066    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-systemd\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285091    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-var-lib-cni-multus\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285116    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-run-multus-certs\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.285156 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285108    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-registration-dir\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-etc-kubernetes\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285186    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-systemd\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285214    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgqcf\" (UniqueName: \"kubernetes.io/projected/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-kube-api-access-hgqcf\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-run-ovn-kubernetes\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285277    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-etc-selinux\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285292    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84f22329-522f-4502-a166-d0f7139138fc-hosts-file\") pod \"node-resolver-ptz2c\" (UID: \"84f22329-522f-4502-a166-d0f7139138fc\") " pod="openshift-dns/node-resolver-ptz2c"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285375    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nxmz\" (UniqueName: \"kubernetes.io/projected/7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5-kube-api-access-8nxmz\") pod \"node-ca-k9gzt\" (UID: \"7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5\") " pod="openshift-image-registry/node-ca-k9gzt"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285389    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-systemd-units\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285403    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-etc-selinux\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285407    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-kubernetes\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-sysctl-d\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285464    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-kubernetes\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285387    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84f22329-522f-4502-a166-d0f7139138fc-hosts-file\") pod \"node-resolver-ptz2c\" (UID: \"84f22329-522f-4502-a166-d0f7139138fc\") " pod="openshift-dns/node-resolver-ptz2c"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285497    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-run-openvswitch\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-env-overrides\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285587    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6542e3b-cdda-4bd2-9419-667e66525978-system-cni-dir\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.285992 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.285607    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285617    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-cni-binary-copy\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285626    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6542e3b-cdda-4bd2-9419-667e66525978-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285647    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-run-openvswitch\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285644    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bd78a31-4667-420c-9ed6-52df573dbaf5-agent-certs\") pod \"konnectivity-agent-tjrpn\" (UID: \"4bd78a31-4667-420c-9ed6-52df573dbaf5\") " pod="kube-system/konnectivity-agent-tjrpn"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285715    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6542e3b-cdda-4bd2-9419-667e66525978-system-cni-dir\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285650    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-sysctl-d\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285784    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-host\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285787    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-systemd-units\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-kubelet\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.285859    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret podName:bdef54cb-b0a3-4783-9655-5296a4f95022 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:56.785843452 +0000 UTC m=+2.982996379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret") pod "global-pull-secret-syncer-jgpsr" (UID: "bdef54cb-b0a3-4783-9655-5296a4f95022") : object "kube-system"/"original-pull-secret" not registered
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-cni-netd\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285924    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-host\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285917    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6542e3b-cdda-4bd2-9419-667e66525978-cnibin\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285306    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-run-ovn-kubernetes\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285932    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-kubelet\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285972    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-cni-netd\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.286822 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.285969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-os-release\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286032    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6542e3b-cdda-4bd2-9419-667e66525978-cnibin\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286036    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-hostroot\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286064    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84f22329-522f-4502-a166-d0f7139138fc-tmp-dir\") pod \"node-resolver-ptz2c\" (UID: \"84f22329-522f-4502-a166-d0f7139138fc\") " pod="openshift-dns/node-resolver-ptz2c"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286086    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdef54cb-b0a3-4783-9655-5296a4f95022-kubelet-config\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286105    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6542e3b-cdda-4bd2-9419-667e66525978-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286136    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-var-lib-kubelet\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286176    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lbgj\" (UniqueName: \"kubernetes.io/projected/49b6a1b2-b8e5-4228-b018-25bc21f3add6-kube-api-access-2lbgj\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286187    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdef54cb-b0a3-4783-9655-5296a4f95022-kubelet-config\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286201    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fnls\" (UniqueName: \"kubernetes.io/projected/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-kube-api-access-6fnls\") pod \"network-metrics-daemon-9pghm\" (UID: \"93029547-ab84-4b4f-ac0f-e1b86f5ca5de\") " pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286373    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-ovnkube-script-lib\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdef54cb-b0a3-4783-9655-5296a4f95022-dbus\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286419    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-run\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286428    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84f22329-522f-4502-a166-d0f7139138fc-tmp-dir\") pod \"node-resolver-ptz2c\" (UID: \"84f22329-522f-4502-a166-d0f7139138fc\") " pod="openshift-dns/node-resolver-ptz2c"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286438    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-sys\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286506    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-run-netns\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.287537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286538    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-var-lib-cni-bin\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286543    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286590    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-env-overrides\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286794    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6395f961-088a-4295-b37d-8e6660b93365-host-slash\") pod \"iptables-alerter-5xcbw\" (UID: \"6395f961-088a-4295-b37d-8e6660b93365\") " pod="openshift-network-operator/iptables-alerter-5xcbw"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286824    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-run-netns\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-multus-cni-dir\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-device-dir\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-host-run-netns\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.286937    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6542e3b-cdda-4bd2-9419-667e66525978-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.287005    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-sys\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.287001    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/087b94d6-aea4-4f87-8c4a-cd199129891d-run\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.287025    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6395f961-088a-4295-b37d-8e6660b93365-host-slash\") pod \"iptables-alerter-5xcbw\" (UID: \"6395f961-088a-4295-b37d-8e6660b93365\") " pod="openshift-network-operator/iptables-alerter-5xcbw"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.287055    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdef54cb-b0a3-4783-9655-5296a4f95022-dbus\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.287079    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49b6a1b2-b8e5-4228-b018-25bc21f3add6-device-dir\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.287402    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-ovnkube-script-lib\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.288487    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/087b94d6-aea4-4f87-8c4a-cd199129891d-tmp\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.288662    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-ovn-node-metrics-cert\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.289482    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bd78a31-4667-420c-9ed6-52df573dbaf5-agent-certs\") pod \"konnectivity-agent-tjrpn\" (UID: \"4bd78a31-4667-420c-9ed6-52df573dbaf5\") " pod="kube-system/konnectivity-agent-tjrpn"
Jul 13 14:52:56.302898 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.289913    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/087b94d6-aea4-4f87-8c4a-cd199129891d-etc-tuned\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.303559 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.293958    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 14:52:56.303559 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.293979    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 14:52:56.303559 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.293992    2564 projected.go:196] Error preparing data for projected volume kube-api-access-j57h2 for pod openshift-network-diagnostics/network-check-target-hl7zq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 14:52:56.303559 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.294187    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2 podName:af9196c9-6c2a-466c-bea6-fd690110ef71 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:56.794164939 +0000 UTC m=+2.991317888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j57h2" (UniqueName: "kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2") pod "network-check-target-hl7zq" (UID: "af9196c9-6c2a-466c-bea6-fd690110ef71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 14:52:56.303559 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.295080    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tb7g\" (UniqueName: \"kubernetes.io/projected/087b94d6-aea4-4f87-8c4a-cd199129891d-kube-api-access-8tb7g\") pod \"tuned-fprbx\" (UID: \"087b94d6-aea4-4f87-8c4a-cd199129891d\") " pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.303559 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.296761    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbfx4\" (UniqueName: \"kubernetes.io/projected/f6542e3b-cdda-4bd2-9419-667e66525978-kube-api-access-rbfx4\") pod \"multus-additional-cni-plugins-wgw48\" (UID: \"f6542e3b-cdda-4bd2-9419-667e66525978\") " pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.303559 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.296977    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pknt5\" (UniqueName: \"kubernetes.io/projected/84f22329-522f-4502-a166-d0f7139138fc-kube-api-access-pknt5\") pod \"node-resolver-ptz2c\" (UID: \"84f22329-522f-4502-a166-d0f7139138fc\") " pod="openshift-dns/node-resolver-ptz2c"
Jul 13 14:52:56.303559 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.299449    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mqj5\" (UniqueName: \"kubernetes.io/projected/8ffb49a6-e992-4bd6-a0ee-2e9c41258047-kube-api-access-5mqj5\") pod \"ovnkube-node-tfgpx\" (UID: \"8ffb49a6-e992-4bd6-a0ee-2e9c41258047\") " pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.303559 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.300359    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lbgj\" (UniqueName: \"kubernetes.io/projected/49b6a1b2-b8e5-4228-b018-25bc21f3add6-kube-api-access-2lbgj\") pod \"aws-ebs-csi-driver-node-wz6fk\" (UID: \"49b6a1b2-b8e5-4228-b018-25bc21f3add6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.303559 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.301893    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td4xb\" (UniqueName: \"kubernetes.io/projected/6395f961-088a-4295-b37d-8e6660b93365-kube-api-access-td4xb\") pod \"iptables-alerter-5xcbw\" (UID: \"6395f961-088a-4295-b37d-8e6660b93365\") " pod="openshift-network-operator/iptables-alerter-5xcbw"
Jul 13 14:52:56.303559 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.301993    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nxmz\" (UniqueName: \"kubernetes.io/projected/7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5-kube-api-access-8nxmz\") pod \"node-ca-k9gzt\" (UID: \"7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5\") " pod="openshift-image-registry/node-ca-k9gzt"
Jul 13 14:52:56.303559 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.302126    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fnls\" (UniqueName: \"kubernetes.io/projected/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-kube-api-access-6fnls\") pod \"network-metrics-daemon-9pghm\" (UID: \"93029547-ab84-4b4f-ac0f-e1b86f5ca5de\") " pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:52:56.315026 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.314964    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal" event={"ID":"4560c566f29ffa1717b75b3bb741078b","Type":"ContainerStarted","Data":"8f6990f82bec2f8e208b2fe95c26990c180d0dcfd378b67a2bb5619470d2392e"}
Jul 13 14:52:56.387666 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387632    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-os-release\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.387844 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387678    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-hostroot\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.387844 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387723    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-var-lib-kubelet\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.387844 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-run-netns\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.387844 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387767    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-var-lib-cni-bin\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.387844 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387789    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-multus-cni-dir\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.387844 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387793    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-hostroot\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.387844 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387803    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-os-release\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.387844 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387847    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-cnibin\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387851    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-run-netns\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387859    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-multus-cni-dir\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387864    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-multus-conf-dir\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387891    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-var-lib-kubelet\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387905    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-cnibin\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387917    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-multus-conf-dir\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387920    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-system-cni-dir\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387891    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-system-cni-dir\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387940    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-multus-daemon-config\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387950    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-var-lib-cni-bin\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.387989    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-run-k8s-cni-cncf-io\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.388015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-multus-socket-dir-parent\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.388060    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-var-lib-cni-multus\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.388059    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-run-k8s-cni-cncf-io\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.388084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-run-multus-certs\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.388108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-etc-kubernetes\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.388124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-var-lib-cni-multus\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388324 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.388133    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgqcf\" (UniqueName: \"kubernetes.io/projected/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-kube-api-access-hgqcf\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388977 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.388128    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-multus-socket-dir-parent\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388977 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.388170    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-host-run-multus-certs\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388977 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.388190    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-etc-kubernetes\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388977 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.388253    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-cni-binary-copy\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388977 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.388566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-multus-daemon-config\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.388977 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.388756    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-cni-binary-copy\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.398415 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.398347    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgqcf\" (UniqueName: \"kubernetes.io/projected/87f8f1a1-06b4-46a3-9e62-adc4d80241a9-kube-api-access-hgqcf\") pod \"multus-rk6vg\" (UID: \"87f8f1a1-06b4-46a3-9e62-adc4d80241a9\") " pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.471575 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.471558    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k9gzt"
Jul 13 14:52:56.478993 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.478970    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tjrpn"
Jul 13 14:52:56.487271 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:56.487245    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd78a31_4667_420c_9ed6_52df573dbaf5.slice/crio-6fc60ae89b890efddbf82735ddeacdc48be7f63229ff40560d37e9ad6fc6d8ac WatchSource:0}: Error finding container 6fc60ae89b890efddbf82735ddeacdc48be7f63229ff40560d37e9ad6fc6d8ac: Status 404 returned error can't find the container with id 6fc60ae89b890efddbf82735ddeacdc48be7f63229ff40560d37e9ad6fc6d8ac
Jul 13 14:52:56.491018 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.490998    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk"
Jul 13 14:52:56.494710 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.494677    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fprbx"
Jul 13 14:52:56.501779 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.501760    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:52:56.502088 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:56.502059    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b6a1b2_b8e5_4228_b018_25bc21f3add6.slice/crio-55ca67c8ec06c60e69fb535e224f021d5ebf27274f42f0d58ac126f108e05ebb WatchSource:0}: Error finding container 55ca67c8ec06c60e69fb535e224f021d5ebf27274f42f0d58ac126f108e05ebb: Status 404 returned error can't find the container with id 55ca67c8ec06c60e69fb535e224f021d5ebf27274f42f0d58ac126f108e05ebb
Jul 13 14:52:56.506359 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:56.506330    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087b94d6_aea4_4f87_8c4a_cd199129891d.slice/crio-152df6b5a800bb762fc4a1288c9b6d90b0520c01ea9dcd243b9813154aa15ad0 WatchSource:0}: Error finding container 152df6b5a800bb762fc4a1288c9b6d90b0520c01ea9dcd243b9813154aa15ad0: Status 404 returned error can't find the container with id 152df6b5a800bb762fc4a1288c9b6d90b0520c01ea9dcd243b9813154aa15ad0
Jul 13 14:52:56.509618 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.509598    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ptz2c"
Jul 13 14:52:56.510648 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:56.510610    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffb49a6_e992_4bd6_a0ee_2e9c41258047.slice/crio-271d3cb4032dcb2f75c443bc48320646df910fbbb4eabb718e0d2bf13d42ae7b WatchSource:0}: Error finding container 271d3cb4032dcb2f75c443bc48320646df910fbbb4eabb718e0d2bf13d42ae7b: Status 404 returned error can't find the container with id 271d3cb4032dcb2f75c443bc48320646df910fbbb4eabb718e0d2bf13d42ae7b
Jul 13 14:52:56.517328 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:56.517306    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f22329_522f_4502_a166_d0f7139138fc.slice/crio-b80e4679d3eed3dfd0c70e307fe05bfbdf0f59dd2ef6b6588cd17291887f904c WatchSource:0}: Error finding container b80e4679d3eed3dfd0c70e307fe05bfbdf0f59dd2ef6b6588cd17291887f904c: Status 404 returned error can't find the container with id b80e4679d3eed3dfd0c70e307fe05bfbdf0f59dd2ef6b6588cd17291887f904c
Jul 13 14:52:56.519040 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.519019    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5xcbw"
Jul 13 14:52:56.524798 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.524776    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wgw48"
Jul 13 14:52:56.529935 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:56.529907    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6395f961_088a_4295_b37d_8e6660b93365.slice/crio-66ee466e7cca34ec358c16eac2a457467c7c3df367d0b5fd704be885377ef8c5 WatchSource:0}: Error finding container 66ee466e7cca34ec358c16eac2a457467c7c3df367d0b5fd704be885377ef8c5: Status 404 returned error can't find the container with id 66ee466e7cca34ec358c16eac2a457467c7c3df367d0b5fd704be885377ef8c5
Jul 13 14:52:56.530060 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.530045    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rk6vg"
Jul 13 14:52:56.533462 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:56.533442    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6542e3b_cdda_4bd2_9419_667e66525978.slice/crio-3c715980c1e82e4bcf91a12d7543c7b1ba97b984639e7b96a2e063587877695d WatchSource:0}: Error finding container 3c715980c1e82e4bcf91a12d7543c7b1ba97b984639e7b96a2e063587877695d: Status 404 returned error can't find the container with id 3c715980c1e82e4bcf91a12d7543c7b1ba97b984639e7b96a2e063587877695d
Jul 13 14:52:56.538238 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:52:56.538162    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f8f1a1_06b4_46a3_9e62_adc4d80241a9.slice/crio-3e6e000a27e70b7a0303207659852d5071b0994c678d4613de0d94c6528d5526 WatchSource:0}: Error finding container 3e6e000a27e70b7a0303207659852d5071b0994c678d4613de0d94c6528d5526: Status 404 returned error can't find the container with id 3e6e000a27e70b7a0303207659852d5071b0994c678d4613de0d94c6528d5526
Jul 13 14:52:56.660891 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.660758    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 14:52:56.790720 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.790666    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:52:56.790871 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.790772    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs\") pod \"network-metrics-daemon-9pghm\" (UID: \"93029547-ab84-4b4f-ac0f-e1b86f5ca5de\") " pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:52:56.790871 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.790820    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 14:52:56.790937 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.790870    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 14:52:56.790937 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.790883    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret podName:bdef54cb-b0a3-4783-9655-5296a4f95022 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:57.790865934 +0000 UTC m=+3.988018881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret") pod "global-pull-secret-syncer-jgpsr" (UID: "bdef54cb-b0a3-4783-9655-5296a4f95022") : object "kube-system"/"original-pull-secret" not registered
Jul 13 14:52:56.790937 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.790916    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs podName:93029547-ab84-4b4f-ac0f-e1b86f5ca5de nodeName:}" failed. No retries permitted until 2026-07-13 14:52:57.790903038 +0000 UTC m=+3.988055965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs") pod "network-metrics-daemon-9pghm" (UID: "93029547-ab84-4b4f-ac0f-e1b86f5ca5de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 14:52:56.891767 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:56.891740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57h2\" (UniqueName: \"kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2\") pod \"network-check-target-hl7zq\" (UID: \"af9196c9-6c2a-466c-bea6-fd690110ef71\") " pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:52:56.891920 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.891896    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 14:52:56.891920 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.891914    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 14:52:56.891999 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.891924    2564 projected.go:196] Error preparing data for projected volume kube-api-access-j57h2 for pod openshift-network-diagnostics/network-check-target-hl7zq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 14:52:56.891999 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:56.891972    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2 podName:af9196c9-6c2a-466c-bea6-fd690110ef71 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:57.891958354 +0000 UTC m=+4.089111279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j57h2" (UniqueName: "kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2") pod "network-check-target-hl7zq" (UID: "af9196c9-6c2a-466c-bea6-fd690110ef71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 14:52:57.219685 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.219618    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 14:47:56 +0000 UTC" deadline="2028-04-24 09:37:55.680362658 +0000 UTC"
Jul 13 14:52:57.219685 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.219683    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15618h44m58.460684418s"
Jul 13 14:52:57.322734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.321982    2564 generic.go:356] "Generic (PLEG): container finished" podID="4560c566f29ffa1717b75b3bb741078b" containerID="f4fcbe915562cadbc24873a035b43cb6d256a8129d247430ce0865676aac8a8c" exitCode=0
Jul 13 14:52:57.322734 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.322074    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal" event={"ID":"4560c566f29ffa1717b75b3bb741078b","Type":"ContainerDied","Data":"f4fcbe915562cadbc24873a035b43cb6d256a8129d247430ce0865676aac8a8c"}
Jul 13 14:52:57.326271 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.325667    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal" event={"ID":"9fe4ee3c51c80837e4836133e719eda0","Type":"ContainerStarted","Data":"c2e21c59ceae7ba639cacba37843dbf85844b1d89a1f3de6d30914249a563816"}
Jul 13 14:52:57.351051 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.350855    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx" event={"ID":"8ffb49a6-e992-4bd6-a0ee-2e9c41258047","Type":"ContainerStarted","Data":"271d3cb4032dcb2f75c443bc48320646df910fbbb4eabb718e0d2bf13d42ae7b"}
Jul 13 14:52:57.355908 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.355686    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk" event={"ID":"49b6a1b2-b8e5-4228-b018-25bc21f3add6","Type":"ContainerStarted","Data":"55ca67c8ec06c60e69fb535e224f021d5ebf27274f42f0d58ac126f108e05ebb"}
Jul 13 14:52:57.363724 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.363659    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tjrpn" event={"ID":"4bd78a31-4667-420c-9ed6-52df573dbaf5","Type":"ContainerStarted","Data":"6fc60ae89b890efddbf82735ddeacdc48be7f63229ff40560d37e9ad6fc6d8ac"}
Jul 13 14:52:57.387077 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.386983    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k9gzt" event={"ID":"7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5","Type":"ContainerStarted","Data":"37db028eb34b22c2318aeb1ece16fcd240af24086079dd2a6b65c6c66d6fb863"}
Jul 13 14:52:57.394279 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.392147    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rk6vg" event={"ID":"87f8f1a1-06b4-46a3-9e62-adc4d80241a9","Type":"ContainerStarted","Data":"3e6e000a27e70b7a0303207659852d5071b0994c678d4613de0d94c6528d5526"}
Jul 13 14:52:57.398992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.398963    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgw48" event={"ID":"f6542e3b-cdda-4bd2-9419-667e66525978","Type":"ContainerStarted","Data":"3c715980c1e82e4bcf91a12d7543c7b1ba97b984639e7b96a2e063587877695d"}
Jul 13 14:52:57.412751 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.408426    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5xcbw" event={"ID":"6395f961-088a-4295-b37d-8e6660b93365","Type":"ContainerStarted","Data":"66ee466e7cca34ec358c16eac2a457467c7c3df367d0b5fd704be885377ef8c5"}
Jul 13 14:52:57.412751 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.411438    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ptz2c" event={"ID":"84f22329-522f-4502-a166-d0f7139138fc","Type":"ContainerStarted","Data":"b80e4679d3eed3dfd0c70e307fe05bfbdf0f59dd2ef6b6588cd17291887f904c"}
Jul 13 14:52:57.416381 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.416295    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fprbx" event={"ID":"087b94d6-aea4-4f87-8c4a-cd199129891d","Type":"ContainerStarted","Data":"152df6b5a800bb762fc4a1288c9b6d90b0520c01ea9dcd243b9813154aa15ad0"}
Jul 13 14:52:57.799240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.799153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:52:57.799396 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.799250    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs\") pod \"network-metrics-daemon-9pghm\" (UID: \"93029547-ab84-4b4f-ac0f-e1b86f5ca5de\") " pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:52:57.799466 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:57.799399    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 14:52:57.799521 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:57.799467    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs podName:93029547-ab84-4b4f-ac0f-e1b86f5ca5de nodeName:}" failed. No retries permitted until 2026-07-13 14:52:59.799447283 +0000 UTC m=+5.996600225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs") pod "network-metrics-daemon-9pghm" (UID: "93029547-ab84-4b4f-ac0f-e1b86f5ca5de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 14:52:57.799892 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:57.799872    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 14:52:57.799980 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:57.799930    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret podName:bdef54cb-b0a3-4783-9655-5296a4f95022 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:59.799914813 +0000 UTC m=+5.997067742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret") pod "global-pull-secret-syncer-jgpsr" (UID: "bdef54cb-b0a3-4783-9655-5296a4f95022") : object "kube-system"/"original-pull-secret" not registered
Jul 13 14:52:57.900554 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:57.899858    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57h2\" (UniqueName: \"kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2\") pod \"network-check-target-hl7zq\" (UID: \"af9196c9-6c2a-466c-bea6-fd690110ef71\") " pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:52:57.900554 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:57.900031    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 14:52:57.900554 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:57.900057    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 14:52:57.900554 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:57.900071    2564 projected.go:196] Error preparing data for projected volume kube-api-access-j57h2 for pod openshift-network-diagnostics/network-check-target-hl7zq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 14:52:57.900554 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:57.900129    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2 podName:af9196c9-6c2a-466c-bea6-fd690110ef71 nodeName:}" failed. No retries permitted until 2026-07-13 14:52:59.900110534 +0000 UTC m=+6.097263463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j57h2" (UniqueName: "kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2") pod "network-check-target-hl7zq" (UID: "af9196c9-6c2a-466c-bea6-fd690110ef71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 14:52:58.220232 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:58.220157    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 14:47:56 +0000 UTC" deadline="2028-03-14 17:28:15.669273014 +0000 UTC"
Jul 13 14:52:58.220232 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:58.220195    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14642h35m17.449082333s"
Jul 13 14:52:58.312564 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:58.311820    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:52:58.312564 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:58.311961    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:52:58.312564 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:58.312000    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:52:58.312564 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:58.312071    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:52:58.312564 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:58.312124    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:52:58.312564 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:58.312189    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:52:58.435992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:58.435954    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal" event={"ID":"4560c566f29ffa1717b75b3bb741078b","Type":"ContainerStarted","Data":"cd6f711f2047b530658035ea7ef3af5c521648a8ac446acbb1cd4364508e50e8"}
Jul 13 14:52:59.815095 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:59.815058    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs\") pod \"network-metrics-daemon-9pghm\" (UID: \"93029547-ab84-4b4f-ac0f-e1b86f5ca5de\") " pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:52:59.815565 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:59.815121    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:52:59.815565 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:59.815242    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 14:52:59.815565 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:59.815291    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret podName:bdef54cb-b0a3-4783-9655-5296a4f95022 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:03.815276915 +0000 UTC m=+10.012429841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret") pod "global-pull-secret-syncer-jgpsr" (UID: "bdef54cb-b0a3-4783-9655-5296a4f95022") : object "kube-system"/"original-pull-secret" not registered
Jul 13 14:52:59.815759 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:59.815572    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 14:52:59.815759 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:59.815601    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs podName:93029547-ab84-4b4f-ac0f-e1b86f5ca5de nodeName:}" failed. No retries permitted until 2026-07-13 14:53:03.815592362 +0000 UTC m=+10.012745289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs") pod "network-metrics-daemon-9pghm" (UID: "93029547-ab84-4b4f-ac0f-e1b86f5ca5de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 14:52:59.916067 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:52:59.916032    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57h2\" (UniqueName: \"kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2\") pod \"network-check-target-hl7zq\" (UID: \"af9196c9-6c2a-466c-bea6-fd690110ef71\") " pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:52:59.916288 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:59.916269    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 14:52:59.916361 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:59.916298    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 14:52:59.916361 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:59.916311    2564 projected.go:196] Error preparing data for projected volume kube-api-access-j57h2 for pod openshift-network-diagnostics/network-check-target-hl7zq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 14:52:59.916361 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:52:59.916360    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2 podName:af9196c9-6c2a-466c-bea6-fd690110ef71 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:03.916344961 +0000 UTC m=+10.113497887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j57h2" (UniqueName: "kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2") pod "network-check-target-hl7zq" (UID: "af9196c9-6c2a-466c-bea6-fd690110ef71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 14:53:00.312022 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:00.311937    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:00.312022 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:00.311986    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:00.312022 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:00.311986    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:00.312318 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:00.312109    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:00.312318 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:00.312233    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:00.312419 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:00.312328    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:02.313973 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:02.313212    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:02.313973 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:02.313342    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:02.313973 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:02.313715    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:02.313973 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:02.313801    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:02.313973 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:02.313856    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:02.313973 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:02.313920    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:03.850764 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:03.850733    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs\") pod \"network-metrics-daemon-9pghm\" (UID: \"93029547-ab84-4b4f-ac0f-e1b86f5ca5de\") " pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:03.851199 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:03.850776    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:03.851199 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:03.850914    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 14:53:03.851199 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:03.850972    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret podName:bdef54cb-b0a3-4783-9655-5296a4f95022 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:11.85095579 +0000 UTC m=+18.048108717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret") pod "global-pull-secret-syncer-jgpsr" (UID: "bdef54cb-b0a3-4783-9655-5296a4f95022") : object "kube-system"/"original-pull-secret" not registered
Jul 13 14:53:03.851362 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:03.851303    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 14:53:03.851362 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:03.851350    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs podName:93029547-ab84-4b4f-ac0f-e1b86f5ca5de nodeName:}" failed. No retries permitted until 2026-07-13 14:53:11.851335667 +0000 UTC m=+18.048488596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs") pod "network-metrics-daemon-9pghm" (UID: "93029547-ab84-4b4f-ac0f-e1b86f5ca5de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 14:53:03.951526 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:03.951493    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57h2\" (UniqueName: \"kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2\") pod \"network-check-target-hl7zq\" (UID: \"af9196c9-6c2a-466c-bea6-fd690110ef71\") " pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:03.951683 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:03.951632    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 14:53:03.951683 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:03.951647    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 14:53:03.951683 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:03.951656    2564 projected.go:196] Error preparing data for projected volume kube-api-access-j57h2 for pod openshift-network-diagnostics/network-check-target-hl7zq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 14:53:03.951806 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:03.951711    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2 podName:af9196c9-6c2a-466c-bea6-fd690110ef71 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:11.951682904 +0000 UTC m=+18.148835830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j57h2" (UniqueName: "kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2") pod "network-check-target-hl7zq" (UID: "af9196c9-6c2a-466c-bea6-fd690110ef71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 14:53:04.315813 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:04.314727    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:04.315813 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:04.314851    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:04.315813 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:04.315197    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:04.315813 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:04.315306    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:04.315813 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:04.315354    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:04.315813 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:04.315423    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:06.311658 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:06.311222    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:06.311658 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:06.311222    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:06.311658 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:06.311366    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:06.311658 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:06.311337    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:06.311658 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:06.311445    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:06.311658 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:06.311549    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:08.311245 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:08.311220    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:08.311636 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:08.311220    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:08.311636 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:08.311319    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:08.311636 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:08.311223    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:08.311636 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:08.311392    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:08.311636 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:08.311476    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:10.311288 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:10.311251    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:10.311758 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:10.311257    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:10.311758 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:10.311385    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:10.311758 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:10.311264    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:10.311758 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:10.311461    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:10.311758 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:10.311506    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:11.907941 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:11.907888    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:11.908350 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:11.907992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs\") pod \"network-metrics-daemon-9pghm\" (UID: \"93029547-ab84-4b4f-ac0f-e1b86f5ca5de\") " pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:11.908350 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:11.908061    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 14:53:11.908350 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:11.908098    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 14:53:11.908350 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:11.908130    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret podName:bdef54cb-b0a3-4783-9655-5296a4f95022 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:27.908114362 +0000 UTC m=+34.105267291 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret") pod "global-pull-secret-syncer-jgpsr" (UID: "bdef54cb-b0a3-4783-9655-5296a4f95022") : object "kube-system"/"original-pull-secret" not registered
Jul 13 14:53:11.908350 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:11.908147    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs podName:93029547-ab84-4b4f-ac0f-e1b86f5ca5de nodeName:}" failed. No retries permitted until 2026-07-13 14:53:27.908139755 +0000 UTC m=+34.105292681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs") pod "network-metrics-daemon-9pghm" (UID: "93029547-ab84-4b4f-ac0f-e1b86f5ca5de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 14:53:12.008825 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:12.008782    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57h2\" (UniqueName: \"kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2\") pod \"network-check-target-hl7zq\" (UID: \"af9196c9-6c2a-466c-bea6-fd690110ef71\") " pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:12.008998 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:12.008956    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 14:53:12.008998 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:12.008982    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 14:53:12.008998 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:12.008994    2564 projected.go:196] Error preparing data for projected volume kube-api-access-j57h2 for pod openshift-network-diagnostics/network-check-target-hl7zq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 14:53:12.009135 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:12.009062    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2 podName:af9196c9-6c2a-466c-bea6-fd690110ef71 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:28.009044343 +0000 UTC m=+34.206197401 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j57h2" (UniqueName: "kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2") pod "network-check-target-hl7zq" (UID: "af9196c9-6c2a-466c-bea6-fd690110ef71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 14:53:12.310575 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:12.310494    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:12.310761 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:12.310498    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:12.310761 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:12.310645    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:12.310761 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:12.310498    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:12.310761 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:12.310735    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:12.310988 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:12.310794    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:14.311856 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:14.311389    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:14.311856 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:14.311496    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:14.311856 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:14.311495    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:14.311856 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:14.311560    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:14.311856 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:14.311644    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:14.311856 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:14.311718    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:16.311255 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:16.311199    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:16.311852 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:16.311203    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:16.311852 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:16.311353    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:16.311852 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:16.311219    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:16.311852 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:16.311430    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:16.311852 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:16.311519    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:17.479832 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.479448    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k9gzt" event={"ID":"7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5","Type":"ContainerStarted","Data":"dfcb3606a7a2b3fb467f20af7d5ecef6d47420e867f5cdb0d0271479391361b9"}
Jul 13 14:53:17.480858 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.480838    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rk6vg" event={"ID":"87f8f1a1-06b4-46a3-9e62-adc4d80241a9","Type":"ContainerStarted","Data":"9d50921c7924ea4e93d415ebe81518c6332a3176ea812e1a3aa9ff128ad3b9c6"}
Jul 13 14:53:17.482277 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.482253    2564 generic.go:356] "Generic (PLEG): container finished" podID="f6542e3b-cdda-4bd2-9419-667e66525978" containerID="45cb0c93288af20fd46e95c37f600499f97db48153ea47eb3c434ece9cf40ccb" exitCode=0
Jul 13 14:53:17.482351 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.482331    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgw48" event={"ID":"f6542e3b-cdda-4bd2-9419-667e66525978","Type":"ContainerDied","Data":"45cb0c93288af20fd46e95c37f600499f97db48153ea47eb3c434ece9cf40ccb"}
Jul 13 14:53:17.483742 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.483721    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ptz2c" event={"ID":"84f22329-522f-4502-a166-d0f7139138fc","Type":"ContainerStarted","Data":"8afdcd9f061dae56a697297ca5e0105f99d5444ea34f3a51573eb96293d73d2c"}
Jul 13 14:53:17.485263 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.485127    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fprbx" event={"ID":"087b94d6-aea4-4f87-8c4a-cd199129891d","Type":"ContainerStarted","Data":"8784504883f44f5b6ac3cb8e1ef0cc37aed64fe71d4386981f37071b0d2b8ed5"}
Jul 13 14:53:17.486639 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.486441    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal" event={"ID":"9fe4ee3c51c80837e4836133e719eda0","Type":"ContainerStarted","Data":"a94dab83ed7c17454ab1f2dfd7ccd343fd010657dc074a5ae16344477886fbf3"}
Jul 13 14:53:17.492252 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.492230    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 14:53:17.492590 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.492569    2564 generic.go:356] "Generic (PLEG): container finished" podID="8ffb49a6-e992-4bd6-a0ee-2e9c41258047" containerID="01863b2e8586c37713650a29b6919e7e58cdeb5abfbbe5948fe38141dd7d5250" exitCode=1
Jul 13 14:53:17.492651 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.492636    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx" event={"ID":"8ffb49a6-e992-4bd6-a0ee-2e9c41258047","Type":"ContainerStarted","Data":"abff0b7f007f9ac2d675fa84ce8f71731ff03755a3423eea46e4a56295d24fd1"}
Jul 13 14:53:17.492742 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.492729    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx" event={"ID":"8ffb49a6-e992-4bd6-a0ee-2e9c41258047","Type":"ContainerStarted","Data":"f5e7fa60998615d1c06c964e38a799c22cb9340979092754ae0619d210f534e1"}
Jul 13 14:53:17.492795 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.492770    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx" event={"ID":"8ffb49a6-e992-4bd6-a0ee-2e9c41258047","Type":"ContainerStarted","Data":"b8923414a9f841020e316fe2c0a04bef0afcd8e1e43e64621fc2b5735f448331"}
Jul 13 14:53:17.492846 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.492797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx" event={"ID":"8ffb49a6-e992-4bd6-a0ee-2e9c41258047","Type":"ContainerStarted","Data":"20fd358ab1b3629601ca570fa7d52f85fe44e1de81f5011423399a4f19807d30"}
Jul 13 14:53:17.492846 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.492824    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx" event={"ID":"8ffb49a6-e992-4bd6-a0ee-2e9c41258047","Type":"ContainerDied","Data":"01863b2e8586c37713650a29b6919e7e58cdeb5abfbbe5948fe38141dd7d5250"}
Jul 13 14:53:17.492942 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.492851    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx" event={"ID":"8ffb49a6-e992-4bd6-a0ee-2e9c41258047","Type":"ContainerStarted","Data":"0bdba3cb78ef1e9587ff10fa8882657ec78dcd9c721ab75497d4c461974b56fc"}
Jul 13 14:53:17.494132 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.494107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk" event={"ID":"49b6a1b2-b8e5-4228-b018-25bc21f3add6","Type":"ContainerStarted","Data":"5d3b35dc85be869262c563326615c6d66c65a138e34885e36beb0f09b929314b"}
Jul 13 14:53:17.495322 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.495301    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tjrpn" event={"ID":"4bd78a31-4667-420c-9ed6-52df573dbaf5","Type":"ContainerStarted","Data":"b5219560b3431a8c5eb7f26e448d3bfcfe4575f3e5f81c3f41bcd3936ad35a7c"}
Jul 13 14:53:17.503956 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.503913    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k9gzt" podStartSLOduration=3.262930711 podStartE2EDuration="23.503901084s" totalImagesPullingTime="20.240970373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:17.50354301 +0000 UTC m=+23.700695959" watchObservedRunningTime="2026-07-13 14:53:17.503901084 +0000 UTC m=+23.701054029"
Jul 13 14:53:17.504063 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.503990    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal" podStartSLOduration=22.50398479 podStartE2EDuration="22.50398479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:52:58.454175547 +0000 UTC m=+4.651328488" watchObservedRunningTime="2026-07-13 14:53:17.50398479 +0000 UTC m=+23.701137737"
Jul 13 14:53:17.525279 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.525225    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rk6vg" podStartSLOduration=3.253277287 podStartE2EDuration="23.525212198s" totalImagesPullingTime="20.271934911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:17.524996729 +0000 UTC m=+23.722149678" watchObservedRunningTime="2026-07-13 14:53:17.525212198 +0000 UTC m=+23.722365145"
Jul 13 14:53:17.543423 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.543365    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal" podStartSLOduration=22.543343964 podStartE2EDuration="22.543343964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:17.543289323 +0000 UTC m=+23.740442271" watchObservedRunningTime="2026-07-13 14:53:17.543343964 +0000 UTC m=+23.740496912"
Jul 13 14:53:17.559534 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.559473    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tjrpn" podStartSLOduration=3.323669428 podStartE2EDuration="23.55945371s" totalImagesPullingTime="20.235784282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:17.559339693 +0000 UTC m=+23.756492658" watchObservedRunningTime="2026-07-13 14:53:17.55945371 +0000 UTC m=+23.756606660"
Jul 13 14:53:17.575627 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.575581    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ptz2c" podStartSLOduration=3.37073156 podStartE2EDuration="23.575563401s" totalImagesPullingTime="20.204831841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:17.575226171 +0000 UTC m=+23.772379120" watchObservedRunningTime="2026-07-13 14:53:17.575563401 +0000 UTC m=+23.772716350"
Jul 13 14:53:17.598965 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:17.598911    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fprbx" podStartSLOduration=3.361447635 podStartE2EDuration="23.598891573s" totalImagesPullingTime="20.237443938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:17.598513269 +0000 UTC m=+23.795666219" watchObservedRunningTime="2026-07-13 14:53:17.598891573 +0000 UTC m=+23.796044522"
Jul 13 14:53:18.309912 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:18.309889    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 13 14:53:18.310994 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:18.310973    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:18.311085 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:18.310980    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:18.311085 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:18.311068    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:18.311215 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:18.311193    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:18.311273 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:18.311256    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:18.311352 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:18.311333    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:18.499490 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:18.499450    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk" event={"ID":"49b6a1b2-b8e5-4228-b018-25bc21f3add6","Type":"ContainerStarted","Data":"8b5a49efa6fc136ef459eb8bfa0c88bac033d4f48a2dd573e8ae625ce2815e3a"}
Jul 13 14:53:18.501015 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:18.500986    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5xcbw" event={"ID":"6395f961-088a-4295-b37d-8e6660b93365","Type":"ContainerStarted","Data":"6610dcf8a5cc281eca82ca9f510820a7bfcc4a2bac2af6e9c9e57003964c226c"}
Jul 13 14:53:18.517951 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:18.517904    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5xcbw" podStartSLOduration=4.327515658 podStartE2EDuration="24.517888866s" totalImagesPullingTime="20.190373208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:18.517757916 +0000 UTC m=+24.714910864" watchObservedRunningTime="2026-07-13 14:53:18.517888866 +0000 UTC m=+24.715041813"
Jul 13 14:53:19.245295 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:19.245190    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T14:53:18.309906601Z","UUID":"98013e49-eb1f-4c45-b26c-283a8dabf89b","Handler":null,"Name":"","Endpoint":""}
Jul 13 14:53:19.247260 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:19.247240    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 13 14:53:19.247369 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:19.247285    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 13 14:53:19.505739 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:19.505646    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk" event={"ID":"49b6a1b2-b8e5-4228-b018-25bc21f3add6","Type":"ContainerStarted","Data":"39abe7476daef50ef80834e9527e65cf5ab8e916a8b53207879592c4d7bcffb4"}
Jul 13 14:53:19.541348 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:19.541280    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wz6fk" podStartSLOduration=3.051572364 podStartE2EDuration="25.541258193s" totalImagesPullingTime="22.489685829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:19.540845389 +0000 UTC m=+25.737998339" watchObservedRunningTime="2026-07-13 14:53:19.541258193 +0000 UTC m=+25.738411121"
Jul 13 14:53:20.246830 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:20.246605    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ptz2c_84f22329-522f-4502-a166-d0f7139138fc/dns-node-resolver/0.log"
Jul 13 14:53:20.311307 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:20.311266    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:20.311487 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:20.311270    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:20.311487 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:20.311404    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:20.311487 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:20.311406    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:20.311660 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:20.311497    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:20.311660 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:20.311587    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:20.512118 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:20.512049    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 14:53:20.512503 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:20.512480    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx" event={"ID":"8ffb49a6-e992-4bd6-a0ee-2e9c41258047","Type":"ContainerStarted","Data":"0bd13dc62036fc7319f7e619810fdd847b83aa22b539ad0d7d651d9b263f17b6"}
Jul 13 14:53:21.217380 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:21.217355    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k9gzt_7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5/node-ca/0.log"
Jul 13 14:53:21.342152 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:21.342114    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tjrpn"
Jul 13 14:53:21.342994 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:21.342974    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tjrpn"
Jul 13 14:53:21.515261 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:21.515165    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tjrpn"
Jul 13 14:53:21.515735 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:21.515721    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tjrpn"
Jul 13 14:53:22.310796 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:22.310677    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:22.310796 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:22.310733    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:22.311012 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:22.310688    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:22.311012 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:22.310876    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:22.311121 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:22.311004    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:22.311121 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:22.311099    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:23.523816 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:23.523612    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 14:53:23.524428 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:23.524289    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx" event={"ID":"8ffb49a6-e992-4bd6-a0ee-2e9c41258047","Type":"ContainerStarted","Data":"04a9b35ce1adc092488da2d5cdd29fae5cd8500a733dedf8c05604236ce1223f"}
Jul 13 14:53:23.524510 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:23.524493    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:53:23.524579 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:23.524567    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:53:23.524636 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:23.524629    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:53:23.524782 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:23.524765    2564 scope.go:117] "RemoveContainer" containerID="01863b2e8586c37713650a29b6919e7e58cdeb5abfbbe5948fe38141dd7d5250"
Jul 13 14:53:23.544287 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:23.543915    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:53:23.544287 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:23.544066    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:53:24.311944 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:24.311906    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:24.312165 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:24.312011    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:24.312165 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:24.312053    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:24.312165 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:24.312058    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:24.312165 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:24.312127    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:24.312303 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:24.312211    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:24.528405 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:24.528370    2564 generic.go:356] "Generic (PLEG): container finished" podID="f6542e3b-cdda-4bd2-9419-667e66525978" containerID="a4bb252e6fc3d85b46b4f59ee43fbef67f578921a266cf17e6d6fbf90d9c6c52" exitCode=0
Jul 13 14:53:24.529078 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:24.528457    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgw48" event={"ID":"f6542e3b-cdda-4bd2-9419-667e66525978","Type":"ContainerDied","Data":"a4bb252e6fc3d85b46b4f59ee43fbef67f578921a266cf17e6d6fbf90d9c6c52"}
Jul 13 14:53:24.532491 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:24.532472    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 14:53:24.532930 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:24.532907    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx" event={"ID":"8ffb49a6-e992-4bd6-a0ee-2e9c41258047","Type":"ContainerStarted","Data":"ae4dbfd361e707597db1e54064464e152bf54ef272a0da39526604d13347e969"}
Jul 13 14:53:26.311441 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:26.311406    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:26.311441 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:26.311427    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:26.311992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:26.311525    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:26.311992 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:26.311532    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:26.311992 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:26.311598    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:26.311992 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:26.311644    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:26.538957 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:26.538924    2564 generic.go:356] "Generic (PLEG): container finished" podID="f6542e3b-cdda-4bd2-9419-667e66525978" containerID="fe7087e58db948c6ab6d5c269bea905b3f221f24aed473560f637a2ca0da1937" exitCode=0
Jul 13 14:53:26.539134 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:26.538972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgw48" event={"ID":"f6542e3b-cdda-4bd2-9419-667e66525978","Type":"ContainerDied","Data":"fe7087e58db948c6ab6d5c269bea905b3f221f24aed473560f637a2ca0da1937"}
Jul 13 14:53:26.564103 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:26.564034    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx" podStartSLOduration=12.260476616 podStartE2EDuration="32.564019965s" totalImagesPullingTime="20.303543349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:24.617909095 +0000 UTC m=+30.815062042" watchObservedRunningTime="2026-07-13 14:53:26.564019965 +0000 UTC m=+32.761172913"
Jul 13 14:53:27.922802 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:27.922773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:27.923178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:27.922835    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs\") pod \"network-metrics-daemon-9pghm\" (UID: \"93029547-ab84-4b4f-ac0f-e1b86f5ca5de\") " pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:27.923178 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:27.922910    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 14:53:27.923178 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:27.922910    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 14:53:27.923178 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:27.922960    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs podName:93029547-ab84-4b4f-ac0f-e1b86f5ca5de nodeName:}" failed. No retries permitted until 2026-07-13 14:53:59.922948156 +0000 UTC m=+66.120101082 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs") pod "network-metrics-daemon-9pghm" (UID: "93029547-ab84-4b4f-ac0f-e1b86f5ca5de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 14:53:27.923178 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:27.922972    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret podName:bdef54cb-b0a3-4783-9655-5296a4f95022 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:59.922966845 +0000 UTC m=+66.120119771 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret") pod "global-pull-secret-syncer-jgpsr" (UID: "bdef54cb-b0a3-4783-9655-5296a4f95022") : object "kube-system"/"original-pull-secret" not registered
Jul 13 14:53:28.023379 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:28.023345    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57h2\" (UniqueName: \"kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2\") pod \"network-check-target-hl7zq\" (UID: \"af9196c9-6c2a-466c-bea6-fd690110ef71\") " pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:28.023533 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:28.023502    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 14:53:28.023533 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:28.023521    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 14:53:28.023533 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:28.023530    2564 projected.go:196] Error preparing data for projected volume kube-api-access-j57h2 for pod openshift-network-diagnostics/network-check-target-hl7zq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 14:53:28.023633 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:28.023584    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2 podName:af9196c9-6c2a-466c-bea6-fd690110ef71 nodeName:}" failed. No retries permitted until 2026-07-13 14:54:00.023569512 +0000 UTC m=+66.220722438 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j57h2" (UniqueName: "kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2") pod "network-check-target-hl7zq" (UID: "af9196c9-6c2a-466c-bea6-fd690110ef71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 14:53:28.311158 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:28.311061    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:28.311158 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:28.311070    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:28.311374 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:28.311197    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:28.311374 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:28.311070    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:28.311374 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:28.311241    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:28.311374 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:28.311302    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:28.545955 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:28.545924    2564 generic.go:356] "Generic (PLEG): container finished" podID="f6542e3b-cdda-4bd2-9419-667e66525978" containerID="d50a6bc6b0090f91f27009733f7ca45dd827f54a9e3141e77af3b71e052832ee" exitCode=0
Jul 13 14:53:28.546151 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:28.546000    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgw48" event={"ID":"f6542e3b-cdda-4bd2-9419-667e66525978","Type":"ContainerDied","Data":"d50a6bc6b0090f91f27009733f7ca45dd827f54a9e3141e77af3b71e052832ee"}
Jul 13 14:53:30.310880 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:30.310848    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:30.311302 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:30.310854    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:30.311302 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:30.310982    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:30.311302 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:30.310991    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:30.311302 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:30.311068    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:30.311302 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:30.311159    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:32.311116 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:32.311080    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:32.311116 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:32.311133    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:32.311625 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:32.311079    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:32.311625 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:32.311243    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:32.311625 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:32.311325    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:32.311625 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:32.311399    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:33.560856 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:33.560820    2564 generic.go:356] "Generic (PLEG): container finished" podID="f6542e3b-cdda-4bd2-9419-667e66525978" containerID="e9c96b39de44a506257e2e51f9107d86321e8c56f5dccd4d1876885bd2403e11" exitCode=0
Jul 13 14:53:33.561222 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:33.560898    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgw48" event={"ID":"f6542e3b-cdda-4bd2-9419-667e66525978","Type":"ContainerDied","Data":"e9c96b39de44a506257e2e51f9107d86321e8c56f5dccd4d1876885bd2403e11"}
Jul 13 14:53:34.311562 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:34.311381    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:34.311737 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:34.311464    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:34.311737 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:34.311494    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:34.311807 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:34.311754    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:34.311807 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:34.311624    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:34.311873 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:34.311807    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:34.565666 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:34.565565    2564 generic.go:356] "Generic (PLEG): container finished" podID="f6542e3b-cdda-4bd2-9419-667e66525978" containerID="9882521e69d2068c5d41169e021b1a490286f60ca27829b8a4da114918f3fd0d" exitCode=0
Jul 13 14:53:34.565666 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:34.565651    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgw48" event={"ID":"f6542e3b-cdda-4bd2-9419-667e66525978","Type":"ContainerDied","Data":"9882521e69d2068c5d41169e021b1a490286f60ca27829b8a4da114918f3fd0d"}
Jul 13 14:53:35.571461 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:35.571428    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wgw48" event={"ID":"f6542e3b-cdda-4bd2-9419-667e66525978","Type":"ContainerStarted","Data":"d8b7217be25bd61d208eaf79e8312a68361889e99c5c7af4fa25f8b485684b48"}
Jul 13 14:53:35.594812 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:35.594769    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wgw48" podStartSLOduration=8.605562931 podStartE2EDuration="41.594754914s" totalImagesPullingTime="32.989191983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:35.59456264 +0000 UTC m=+41.791715590" watchObservedRunningTime="2026-07-13 14:53:35.594754914 +0000 UTC m=+41.791907858"
Jul 13 14:53:36.310727 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:36.310675    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:36.310727 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:36.310710    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:36.310727 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:36.310730    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:36.310948 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:36.310814    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:36.310948 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:36.310883    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:36.311015 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:36.310937    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:38.311506 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:38.311476    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:38.311506 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:38.311489    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:38.311941 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:38.311489    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:38.311941 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:38.311565    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:38.311941 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:38.311635    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:38.311941 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:38.311731    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:40.311057 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:40.311026    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:40.311459 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:40.311024    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:40.311459 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:40.311137    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:40.311459 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:40.311031    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:40.311459 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:40.311292    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:40.311459 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:40.311408    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:42.311257 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:42.311217    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:42.311788 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:42.311225    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:42.311788 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:42.311336    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:42.311788 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:42.311402    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:42.311788 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:42.311225    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:42.311788 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:42.311468    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:44.313908 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:44.313879    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:44.314266 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:44.313880    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:44.314266 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:44.313880    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:44.314266 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:44.314058    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:44.314266 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:44.314134    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:44.314266 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:44.313968    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:45.170892 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:45.170858    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jgpsr"]
Jul 13 14:53:45.171059 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:45.170982    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:45.171117 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:45.171095    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:45.171438 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:45.171416    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hl7zq"]
Jul 13 14:53:45.171536 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:45.171495    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:45.171602 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:45.171570    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:45.172225 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:45.172202    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9pghm"]
Jul 13 14:53:45.172295 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:45.172283    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:45.172383 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:45.172364    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:47.311301 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:47.311268    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:47.312025 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:47.311361    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:47.312025 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:47.311368    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:47.312025 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:47.311461    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:47.312025 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:47.311495    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:47.312025 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:47.311547    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:49.311019 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.310836    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:49.311392 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.310893    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:49.311392 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:49.311086    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hl7zq" podUID="af9196c9-6c2a-466c-bea6-fd690110ef71"
Jul 13 14:53:49.311392 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.310893    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:49.311392 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:49.311162    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jgpsr" podUID="bdef54cb-b0a3-4783-9655-5296a4f95022"
Jul 13 14:53:49.311392 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:49.311228    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9pghm" podUID="93029547-ab84-4b4f-ac0f-e1b86f5ca5de"
Jul 13 14:53:49.653737 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.653715    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeReady"
Jul 13 14:53:49.653882 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.653837    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 13 14:53:49.743556 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.743516    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4dd6p"]
Jul 13 14:53:49.745679 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.745658    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:49.745826 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.745754    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5xl9h"]
Jul 13 14:53:49.747657 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.747642    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5xl9h"
Jul 13 14:53:49.756090 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.756072    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 13 14:53:49.756182 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.756093    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 13 14:53:49.759506 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.759485    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4s4js\""
Jul 13 14:53:49.760006 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.759991    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5mgh5\""
Jul 13 14:53:49.760743 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.760729    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 13 14:53:49.763560 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.763546    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 13 14:53:49.764437 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.764421    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 13 14:53:49.776809 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.776791    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4dd6p"]
Jul 13 14:53:49.782431 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.782413    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxmlj\" (UniqueName: \"kubernetes.io/projected/33636c73-08ef-4a53-9365-76554fce2144-kube-api-access-gxmlj\") pod \"dns-default-4dd6p\" (UID: \"33636c73-08ef-4a53-9365-76554fce2144\") " pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:49.782517 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.782459    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/33636c73-08ef-4a53-9365-76554fce2144-config-volume\") pod \"dns-default-4dd6p\" (UID: \"33636c73-08ef-4a53-9365-76554fce2144\") " pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:49.782517 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.782485    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b085a1b-e5a1-4106-a6a3-e7ada1a20f82-cert\") pod \"ingress-canary-5xl9h\" (UID: \"7b085a1b-e5a1-4106-a6a3-e7ada1a20f82\") " pod="openshift-ingress-canary/ingress-canary-5xl9h"
Jul 13 14:53:49.782588 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.782527    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75m6n\" (UniqueName: \"kubernetes.io/projected/7b085a1b-e5a1-4106-a6a3-e7ada1a20f82-kube-api-access-75m6n\") pod \"ingress-canary-5xl9h\" (UID: \"7b085a1b-e5a1-4106-a6a3-e7ada1a20f82\") " pod="openshift-ingress-canary/ingress-canary-5xl9h"
Jul 13 14:53:49.782621 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.782588    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33636c73-08ef-4a53-9365-76554fce2144-tmp-dir\") pod \"dns-default-4dd6p\" (UID: \"33636c73-08ef-4a53-9365-76554fce2144\") " pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:49.782621 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.782605    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33636c73-08ef-4a53-9365-76554fce2144-metrics-tls\") pod \"dns-default-4dd6p\" (UID: \"33636c73-08ef-4a53-9365-76554fce2144\") " pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:49.784112 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.784096    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5xl9h"]
Jul 13 14:53:49.785537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.785518    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nf6sd"]
Jul 13 14:53:49.787894 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.787875    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.793195 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.793168    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 13 14:53:49.793291 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.793216    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 13 14:53:49.793347 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.793329    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 13 14:53:49.793458 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.793445    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ghzxf\""
Jul 13 14:53:49.793974 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.793961    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 13 14:53:49.810723 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.810683    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nf6sd"]
Jul 13 14:53:49.883851 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.883827    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b085a1b-e5a1-4106-a6a3-e7ada1a20f82-cert\") pod \"ingress-canary-5xl9h\" (UID: \"7b085a1b-e5a1-4106-a6a3-e7ada1a20f82\") " pod="openshift-ingress-canary/ingress-canary-5xl9h"
Jul 13 14:53:49.883851 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.883856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75m6n\" (UniqueName: \"kubernetes.io/projected/7b085a1b-e5a1-4106-a6a3-e7ada1a20f82-kube-api-access-75m6n\") pod \"ingress-canary-5xl9h\" (UID: \"7b085a1b-e5a1-4106-a6a3-e7ada1a20f82\") " pod="openshift-ingress-canary/ingress-canary-5xl9h"
Jul 13 14:53:49.884080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.883882    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-crio-socket\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.884080 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.883924    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-data-volume\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.884177 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.884101    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33636c73-08ef-4a53-9365-76554fce2144-tmp-dir\") pod \"dns-default-4dd6p\" (UID: \"33636c73-08ef-4a53-9365-76554fce2144\") " pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:49.884177 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.884126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33636c73-08ef-4a53-9365-76554fce2144-metrics-tls\") pod \"dns-default-4dd6p\" (UID: \"33636c73-08ef-4a53-9365-76554fce2144\") " pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:49.884177 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.884145    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.884177 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.884165    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxmlj\" (UniqueName: \"kubernetes.io/projected/33636c73-08ef-4a53-9365-76554fce2144-kube-api-access-gxmlj\") pod \"dns-default-4dd6p\" (UID: \"33636c73-08ef-4a53-9365-76554fce2144\") " pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:49.884341 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.884195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw6rj\" (UniqueName: \"kubernetes.io/projected/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-kube-api-access-qw6rj\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.884341 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.884304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.884341 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.884334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/33636c73-08ef-4a53-9365-76554fce2144-config-volume\") pod \"dns-default-4dd6p\" (UID: \"33636c73-08ef-4a53-9365-76554fce2144\") " pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:49.884471 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.884420    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33636c73-08ef-4a53-9365-76554fce2144-tmp-dir\") pod \"dns-default-4dd6p\" (UID: \"33636c73-08ef-4a53-9365-76554fce2144\") " pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:49.884793 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.884774    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/33636c73-08ef-4a53-9365-76554fce2144-config-volume\") pod \"dns-default-4dd6p\" (UID: \"33636c73-08ef-4a53-9365-76554fce2144\") " pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:49.888036 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.888010    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33636c73-08ef-4a53-9365-76554fce2144-metrics-tls\") pod \"dns-default-4dd6p\" (UID: \"33636c73-08ef-4a53-9365-76554fce2144\") " pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:49.888183 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.888165    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b085a1b-e5a1-4106-a6a3-e7ada1a20f82-cert\") pod \"ingress-canary-5xl9h\" (UID: \"7b085a1b-e5a1-4106-a6a3-e7ada1a20f82\") " pod="openshift-ingress-canary/ingress-canary-5xl9h"
Jul 13 14:53:49.908767 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.908687    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxmlj\" (UniqueName: \"kubernetes.io/projected/33636c73-08ef-4a53-9365-76554fce2144-kube-api-access-gxmlj\") pod \"dns-default-4dd6p\" (UID: \"33636c73-08ef-4a53-9365-76554fce2144\") " pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:49.908869 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.908824    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75m6n\" (UniqueName: \"kubernetes.io/projected/7b085a1b-e5a1-4106-a6a3-e7ada1a20f82-kube-api-access-75m6n\") pod \"ingress-canary-5xl9h\" (UID: \"7b085a1b-e5a1-4106-a6a3-e7ada1a20f82\") " pod="openshift-ingress-canary/ingress-canary-5xl9h"
Jul 13 14:53:49.984963 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.984926    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw6rj\" (UniqueName: \"kubernetes.io/projected/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-kube-api-access-qw6rj\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.984963 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.984965    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.985191 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.985112    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-crio-socket\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.985191 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.985153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-data-volume\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.985298 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.985246    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.985351 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.985307    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-crio-socket\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.985523 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.985503    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-data-volume\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.985821 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.985802    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.987330 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.987310    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:49.996861 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:49.996831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw6rj\" (UniqueName: \"kubernetes.io/projected/c680c3f4-6691-4a7b-ac7b-8fdf893a832d-kube-api-access-qw6rj\") pod \"insights-runtime-extractor-nf6sd\" (UID: \"c680c3f4-6691-4a7b-ac7b-8fdf893a832d\") " pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:50.055879 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:50.055849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:50.060643 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:50.060626    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5xl9h"
Jul 13 14:53:50.097164 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:50.096910    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nf6sd"
Jul 13 14:53:50.227719 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:50.227565    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5xl9h"]
Jul 13 14:53:50.231574 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:53:50.231545    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b085a1b_e5a1_4106_a6a3_e7ada1a20f82.slice/crio-8c028d5fecf1ef26d0cf8fe0c2a6e0d660d2ddee35503cd146d3f34edbbd0f7b WatchSource:0}: Error finding container 8c028d5fecf1ef26d0cf8fe0c2a6e0d660d2ddee35503cd146d3f34edbbd0f7b: Status 404 returned error can't find the container with id 8c028d5fecf1ef26d0cf8fe0c2a6e0d660d2ddee35503cd146d3f34edbbd0f7b
Jul 13 14:53:50.243985 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:50.243960    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4dd6p"]
Jul 13 14:53:50.247254 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:53:50.247228    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33636c73_08ef_4a53_9365_76554fce2144.slice/crio-cfe7823af6379b3bca74e44755be66afe7ad6afaab6d1032e4233e4811d03754 WatchSource:0}: Error finding container cfe7823af6379b3bca74e44755be66afe7ad6afaab6d1032e4233e4811d03754: Status 404 returned error can't find the container with id cfe7823af6379b3bca74e44755be66afe7ad6afaab6d1032e4233e4811d03754
Jul 13 14:53:50.274781 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:50.272787    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nf6sd"]
Jul 13 14:53:50.278058 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:53:50.278038    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc680c3f4_6691_4a7b_ac7b_8fdf893a832d.slice/crio-22325da7f1701bc4aa882f5dc4e5da4e31dfb90bf3f7f494df8b4b15365af408 WatchSource:0}: Error finding container 22325da7f1701bc4aa882f5dc4e5da4e31dfb90bf3f7f494df8b4b15365af408: Status 404 returned error can't find the container with id 22325da7f1701bc4aa882f5dc4e5da4e31dfb90bf3f7f494df8b4b15365af408
Jul 13 14:53:50.604139 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:50.604051    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4dd6p" event={"ID":"33636c73-08ef-4a53-9365-76554fce2144","Type":"ContainerStarted","Data":"cfe7823af6379b3bca74e44755be66afe7ad6afaab6d1032e4233e4811d03754"}
Jul 13 14:53:50.605179 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:50.605150    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nf6sd" event={"ID":"c680c3f4-6691-4a7b-ac7b-8fdf893a832d","Type":"ContainerStarted","Data":"22325da7f1701bc4aa882f5dc4e5da4e31dfb90bf3f7f494df8b4b15365af408"}
Jul 13 14:53:50.606239 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:50.606217    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5xl9h" event={"ID":"7b085a1b-e5a1-4106-a6a3-e7ada1a20f82","Type":"ContainerStarted","Data":"8c028d5fecf1ef26d0cf8fe0c2a6e0d660d2ddee35503cd146d3f34edbbd0f7b"}
Jul 13 14:53:51.311585 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:51.311535    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:51.311997 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:51.311981    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:51.313292 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:51.312424    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:53:51.321731 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:51.321707    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 14:53:51.322444 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:51.321954    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 14:53:51.322808 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:51.322789    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 14:53:51.323059 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:51.323031    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 14:53:51.323211 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:51.323195    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hw4cv\""
Jul 13 14:53:51.323398 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:51.323371    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bz4r7\""
Jul 13 14:53:54.813128 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:54.813090    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-x96cn"]
Jul 13 14:53:54.821039 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:54.821015    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:54.826082 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:54.826060    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4hbgw\""
Jul 13 14:53:54.826973 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:54.826953    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 13 14:53:54.827085 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:54.827001    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 13 14:53:54.827085 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:54.827009    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 13 14:53:54.827311 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:54.827292    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 13 14:53:54.827435 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:54.827375    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 13 14:53:54.828665 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:54.828645    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-x96cn"]
Jul 13 14:53:54.924835 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:54.924796    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88dnw\" (UniqueName: \"kubernetes.io/projected/32357f5b-ef2c-449c-902f-c4db8304ca83-kube-api-access-88dnw\") pod \"prometheus-operator-64785b6675-x96cn\" (UID: \"32357f5b-ef2c-449c-902f-c4db8304ca83\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:54.924980 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:54.924865    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32357f5b-ef2c-449c-902f-c4db8304ca83-metrics-client-ca\") pod \"prometheus-operator-64785b6675-x96cn\" (UID: \"32357f5b-ef2c-449c-902f-c4db8304ca83\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:54.924980 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:54.924914    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32357f5b-ef2c-449c-902f-c4db8304ca83-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-x96cn\" (UID: \"32357f5b-ef2c-449c-902f-c4db8304ca83\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:54.925091 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:54.924990    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/32357f5b-ef2c-449c-902f-c4db8304ca83-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-x96cn\" (UID: \"32357f5b-ef2c-449c-902f-c4db8304ca83\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:55.025566 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.025480    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88dnw\" (UniqueName: \"kubernetes.io/projected/32357f5b-ef2c-449c-902f-c4db8304ca83-kube-api-access-88dnw\") pod \"prometheus-operator-64785b6675-x96cn\" (UID: \"32357f5b-ef2c-449c-902f-c4db8304ca83\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:55.025566 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.025546    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32357f5b-ef2c-449c-902f-c4db8304ca83-metrics-client-ca\") pod \"prometheus-operator-64785b6675-x96cn\" (UID: \"32357f5b-ef2c-449c-902f-c4db8304ca83\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:55.025824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.025603    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32357f5b-ef2c-449c-902f-c4db8304ca83-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-x96cn\" (UID: \"32357f5b-ef2c-449c-902f-c4db8304ca83\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:55.025824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.025654    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/32357f5b-ef2c-449c-902f-c4db8304ca83-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-x96cn\" (UID: \"32357f5b-ef2c-449c-902f-c4db8304ca83\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:55.025824 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:55.025790    2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 13 14:53:55.025972 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:53:55.025852    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32357f5b-ef2c-449c-902f-c4db8304ca83-prometheus-operator-tls podName:32357f5b-ef2c-449c-902f-c4db8304ca83 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:55.525832464 +0000 UTC m=+61.722985395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/32357f5b-ef2c-449c-902f-c4db8304ca83-prometheus-operator-tls") pod "prometheus-operator-64785b6675-x96cn" (UID: "32357f5b-ef2c-449c-902f-c4db8304ca83") : secret "prometheus-operator-tls" not found
Jul 13 14:53:55.026345 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.026317    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32357f5b-ef2c-449c-902f-c4db8304ca83-metrics-client-ca\") pod \"prometheus-operator-64785b6675-x96cn\" (UID: \"32357f5b-ef2c-449c-902f-c4db8304ca83\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:55.028635 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.028605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32357f5b-ef2c-449c-902f-c4db8304ca83-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-x96cn\" (UID: \"32357f5b-ef2c-449c-902f-c4db8304ca83\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:55.035798 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.035769    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88dnw\" (UniqueName: \"kubernetes.io/projected/32357f5b-ef2c-449c-902f-c4db8304ca83-kube-api-access-88dnw\") pod \"prometheus-operator-64785b6675-x96cn\" (UID: \"32357f5b-ef2c-449c-902f-c4db8304ca83\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:55.529395 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.529306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/32357f5b-ef2c-449c-902f-c4db8304ca83-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-x96cn\" (UID: \"32357f5b-ef2c-449c-902f-c4db8304ca83\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:55.531748 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.531719    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/32357f5b-ef2c-449c-902f-c4db8304ca83-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-x96cn\" (UID: \"32357f5b-ef2c-449c-902f-c4db8304ca83\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:55.545420 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.545402    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tfgpx"
Jul 13 14:53:55.622532 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.622501    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4dd6p" event={"ID":"33636c73-08ef-4a53-9365-76554fce2144","Type":"ContainerStarted","Data":"b73200cb5bce4d3540a5328f39345bb16224943a9eeec0b066d5ae64a5c10ed3"}
Jul 13 14:53:55.622719 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.622558    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4dd6p" event={"ID":"33636c73-08ef-4a53-9365-76554fce2144","Type":"ContainerStarted","Data":"7371d973268bc5d5f34f903c15cb2877759b3c759e47728315f9ee224a3985fe"}
Jul 13 14:53:55.622719 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.622620    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:53:55.624011 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.623990    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nf6sd" event={"ID":"c680c3f4-6691-4a7b-ac7b-8fdf893a832d","Type":"ContainerStarted","Data":"bb402abb92989d0e65b20b4dd60d5850bf2ff3251e727ef861656a1588bd5b36"}
Jul 13 14:53:55.625349 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.625327    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5xl9h" event={"ID":"7b085a1b-e5a1-4106-a6a3-e7ada1a20f82","Type":"ContainerStarted","Data":"1c42d21e7ffb88336f47f05c4fbcddedb400123a7c4fe9ae66aaa76fa9176e90"}
Jul 13 14:53:55.649903 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.649862    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4dd6p" podStartSLOduration=1.687877887 podStartE2EDuration="6.649849429s" totalImagesPullingTime="4.961971542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:55.648960001 +0000 UTC m=+61.846112960" watchObservedRunningTime="2026-07-13 14:53:55.649849429 +0000 UTC m=+61.847002354"
Jul 13 14:53:55.689284 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.689228    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5xl9h" podStartSLOduration=1.712249774 podStartE2EDuration="6.68920686s" totalImagesPullingTime="4.976957086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:55.68909158 +0000 UTC m=+61.886244540" watchObservedRunningTime="2026-07-13 14:53:55.68920686 +0000 UTC m=+61.886359811"
Jul 13 14:53:55.733238 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.733214    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn"
Jul 13 14:53:55.865013 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:55.864875    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-x96cn"]
Jul 13 14:53:55.867289 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:53:55.867260    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32357f5b_ef2c_449c_902f_c4db8304ca83.slice/crio-01211a05c2f032bc9b703c7f5c9b0f85e37cdd41fee622271bb2337ac4ed5cac WatchSource:0}: Error finding container 01211a05c2f032bc9b703c7f5c9b0f85e37cdd41fee622271bb2337ac4ed5cac: Status 404 returned error can't find the container with id 01211a05c2f032bc9b703c7f5c9b0f85e37cdd41fee622271bb2337ac4ed5cac
Jul 13 14:53:56.630896 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:56.630855    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nf6sd" event={"ID":"c680c3f4-6691-4a7b-ac7b-8fdf893a832d","Type":"ContainerStarted","Data":"c74fcc61c8a6ba1afaa1e77fedb34a74d9288744bb8626bcc9df14b137cc7276"}
Jul 13 14:53:56.631871 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:56.631849    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn" event={"ID":"32357f5b-ef2c-449c-902f-c4db8304ca83","Type":"ContainerStarted","Data":"01211a05c2f032bc9b703c7f5c9b0f85e37cdd41fee622271bb2337ac4ed5cac"}
Jul 13 14:53:59.644421 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:59.644388    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn" event={"ID":"32357f5b-ef2c-449c-902f-c4db8304ca83","Type":"ContainerStarted","Data":"615f7fd6d1b464fb85908947baa57f85ef14c51433e3bffbe0ad157230c6e47b"}
Jul 13 14:53:59.644908 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:59.644487    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn" event={"ID":"32357f5b-ef2c-449c-902f-c4db8304ca83","Type":"ContainerStarted","Data":"84f437f2d24330f1ec06d07922195aea93594dfa6d8c3a6efa8228a4a1ed21cf"}
Jul 13 14:53:59.646393 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:59.646367    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nf6sd" event={"ID":"c680c3f4-6691-4a7b-ac7b-8fdf893a832d","Type":"ContainerStarted","Data":"16cbbc2826b35f07d92d426afb46cda703e69fa82415485bcb5ef3be9b78b542"}
Jul 13 14:53:59.667292 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:59.667241    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-x96cn" podStartSLOduration=2.317056546 podStartE2EDuration="5.667221283s" totalImagesPullingTime="3.350164737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:59.666574325 +0000 UTC m=+65.863727274" watchObservedRunningTime="2026-07-13 14:53:59.667221283 +0000 UTC m=+65.864374233"
Jul 13 14:53:59.694344 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:59.694300    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nf6sd" podStartSLOduration=1.845700129 podStartE2EDuration="10.694280592s" totalImagesPullingTime="8.848580463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:53:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:59.69375737 +0000 UTC m=+65.890910322" watchObservedRunningTime="2026-07-13 14:53:59.694280592 +0000 UTC m=+65.891433539"
Jul 13 14:53:59.963151 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:59.963052    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:59.963151 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:59.963137    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs\") pod \"network-metrics-daemon-9pghm\" (UID: \"93029547-ab84-4b4f-ac0f-e1b86f5ca5de\") " pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:53:59.966305 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:59.966285    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 14:53:59.966588 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:59.966574    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 14:53:59.975930 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:59.975906    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdef54cb-b0a3-4783-9655-5296a4f95022-original-pull-secret\") pod \"global-pull-secret-syncer-jgpsr\" (UID: \"bdef54cb-b0a3-4783-9655-5296a4f95022\") " pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:53:59.975930 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:53:59.975924    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93029547-ab84-4b4f-ac0f-e1b86f5ca5de-metrics-certs\") pod \"network-metrics-daemon-9pghm\" (UID: \"93029547-ab84-4b4f-ac0f-e1b86f5ca5de\") " pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:54:00.028613 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.028588    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jgpsr"
Jul 13 14:54:00.038997 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.038977    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bz4r7\""
Jul 13 14:54:00.047536 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.047516    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9pghm"
Jul 13 14:54:00.064242 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.064214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57h2\" (UniqueName: \"kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2\") pod \"network-check-target-hl7zq\" (UID: \"af9196c9-6c2a-466c-bea6-fd690110ef71\") " pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:54:00.067220 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.067186    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 14:54:00.078183 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.078122    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 14:54:00.088220 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.088192    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j57h2\" (UniqueName: \"kubernetes.io/projected/af9196c9-6c2a-466c-bea6-fd690110ef71-kube-api-access-j57h2\") pod \"network-check-target-hl7zq\" (UID: \"af9196c9-6c2a-466c-bea6-fd690110ef71\") " pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:54:00.168944 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.168910    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jgpsr"]
Jul 13 14:54:00.171954 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:54:00.171926    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdef54cb_b0a3_4783_9655_5296a4f95022.slice/crio-d222a3a59ba281ac333bb85a4500ecf0650b9c475d1926eca5723b2dfe0d05a9 WatchSource:0}: Error finding container d222a3a59ba281ac333bb85a4500ecf0650b9c475d1926eca5723b2dfe0d05a9: Status 404 returned error can't find the container with id d222a3a59ba281ac333bb85a4500ecf0650b9c475d1926eca5723b2dfe0d05a9
Jul 13 14:54:00.189961 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.189942    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9pghm"]
Jul 13 14:54:00.192977 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:54:00.192954    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93029547_ab84_4b4f_ac0f_e1b86f5ca5de.slice/crio-677a36e2694e829a85f6d406690e26f1b34aeb7be280931000574173874fe4c3 WatchSource:0}: Error finding container 677a36e2694e829a85f6d406690e26f1b34aeb7be280931000574173874fe4c3: Status 404 returned error can't find the container with id 677a36e2694e829a85f6d406690e26f1b34aeb7be280931000574173874fe4c3
Jul 13 14:54:00.347367 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.347297    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hw4cv\""
Jul 13 14:54:00.355439 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.355419    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:54:00.480249 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.480217    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hl7zq"]
Jul 13 14:54:00.483322 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:54:00.483296    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9196c9_6c2a_466c_bea6_fd690110ef71.slice/crio-cbeae03f1570d533a51e462f01ea5759b00dc0d0d0573a57d6e445c55d9cffd1 WatchSource:0}: Error finding container cbeae03f1570d533a51e462f01ea5759b00dc0d0d0573a57d6e445c55d9cffd1: Status 404 returned error can't find the container with id cbeae03f1570d533a51e462f01ea5759b00dc0d0d0573a57d6e445c55d9cffd1
Jul 13 14:54:00.651042 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.650982    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hl7zq" event={"ID":"af9196c9-6c2a-466c-bea6-fd690110ef71","Type":"ContainerStarted","Data":"cbeae03f1570d533a51e462f01ea5759b00dc0d0d0573a57d6e445c55d9cffd1"}
Jul 13 14:54:00.652289 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.652261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jgpsr" event={"ID":"bdef54cb-b0a3-4783-9655-5296a4f95022","Type":"ContainerStarted","Data":"d222a3a59ba281ac333bb85a4500ecf0650b9c475d1926eca5723b2dfe0d05a9"}
Jul 13 14:54:00.653637 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:00.653598    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9pghm" event={"ID":"93029547-ab84-4b4f-ac0f-e1b86f5ca5de","Type":"ContainerStarted","Data":"677a36e2694e829a85f6d406690e26f1b34aeb7be280931000574173874fe4c3"}
Jul 13 14:54:01.250965 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.250903    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"]
Jul 13 14:54:01.253393 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.253371    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:01.259136 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.259112    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 13 14:54:01.259724 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.259683    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kxdfz\""
Jul 13 14:54:01.260303 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.260284    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 13 14:54:01.261306 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.261278    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"]
Jul 13 14:54:01.265300 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.265276    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.273232 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.271181    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-d7q77\""
Jul 13 14:54:01.273232 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.271382    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 13 14:54:01.273232 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.271572    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 13 14:54:01.273232 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.271888    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 13 14:54:01.273232 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.272062    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2mnl5"]
Jul 13 14:54:01.275142 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.275123    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.275526 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.275507    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"]
Jul 13 14:54:01.279596 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.279378    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 13 14:54:01.280140 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.280123    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q98rp\""
Jul 13 14:54:01.280326 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.280138    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 13 14:54:01.286790 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.286766    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"]
Jul 13 14:54:01.287392 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.287374    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 13 14:54:01.375170 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375042    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-587c6\" (UniqueName: \"kubernetes.io/projected/e8f2fc05-e0b7-4131-b42b-41aa944c7679-kube-api-access-587c6\") pod \"openshift-state-metrics-84bd68b896-5w66h\" (UID: \"e8f2fc05-e0b7-4131-b42b-41aa944c7679\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:01.375170 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375085    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8f2fc05-e0b7-4131-b42b-41aa944c7679-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5w66h\" (UID: \"e8f2fc05-e0b7-4131-b42b-41aa944c7679\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:01.375170 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375114    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-tls\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.375170 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375138    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/579a5f61-d453-4327-ab8d-ea6dae5d44c9-sys\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375180    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-textfile\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375225    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftnnj\" (UniqueName: \"kubernetes.io/projected/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-api-access-ftnnj\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375292    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8f2fc05-e0b7-4131-b42b-41aa944c7679-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5w66h\" (UID: \"e8f2fc05-e0b7-4131-b42b-41aa944c7679\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375324    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375366    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5471e5ad-8965-4379-af6a-2cb21fb280e7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375396    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f2fc05-e0b7-4131-b42b-41aa944c7679-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5w66h\" (UID: \"e8f2fc05-e0b7-4131-b42b-41aa944c7679\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375422    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375440    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/579a5f61-d453-4327-ab8d-ea6dae5d44c9-metrics-client-ca\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-wtmp\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375508    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5471e5ad-8965-4379-af6a-2cb21fb280e7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375525    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/579a5f61-d453-4327-ab8d-ea6dae5d44c9-root\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375547    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzmz7\" (UniqueName: \"kubernetes.io/projected/579a5f61-d453-4327-ab8d-ea6dae5d44c9-kube-api-access-rzmz7\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.376551 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.375572    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.476448 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476364    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/579a5f61-d453-4327-ab8d-ea6dae5d44c9-metrics-client-ca\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.476448 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-wtmp\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.476448 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476425    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.476716 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476458    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.476716 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5471e5ad-8965-4379-af6a-2cb21fb280e7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.476716 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476517    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/579a5f61-d453-4327-ab8d-ea6dae5d44c9-root\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.476716 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476548    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzmz7\" (UniqueName: \"kubernetes.io/projected/579a5f61-d453-4327-ab8d-ea6dae5d44c9-kube-api-access-rzmz7\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.476716 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476576    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.476716 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476590    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-wtmp\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.476716 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476612    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-587c6\" (UniqueName: \"kubernetes.io/projected/e8f2fc05-e0b7-4131-b42b-41aa944c7679-kube-api-access-587c6\") pod \"openshift-state-metrics-84bd68b896-5w66h\" (UID: \"e8f2fc05-e0b7-4131-b42b-41aa944c7679\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:01.476716 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476661    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8f2fc05-e0b7-4131-b42b-41aa944c7679-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5w66h\" (UID: \"e8f2fc05-e0b7-4131-b42b-41aa944c7679\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:01.476716 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476710    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-tls\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.477125 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/579a5f61-d453-4327-ab8d-ea6dae5d44c9-sys\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.477125 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476764    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-textfile\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.477125 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476787    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftnnj\" (UniqueName: \"kubernetes.io/projected/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-api-access-ftnnj\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.477125 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476827    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8f2fc05-e0b7-4131-b42b-41aa944c7679-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5w66h\" (UID: \"e8f2fc05-e0b7-4131-b42b-41aa944c7679\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:01.477125 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476855    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.477125 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476885    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5471e5ad-8965-4379-af6a-2cb21fb280e7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.477125 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476906    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f2fc05-e0b7-4131-b42b-41aa944c7679-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5w66h\" (UID: \"e8f2fc05-e0b7-4131-b42b-41aa944c7679\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:01.477125 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.476928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.477125 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:54:01.477014    2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 13 14:54:01.477125 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:54:01.477056    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-tls podName:5471e5ad-8965-4379-af6a-2cb21fb280e7 nodeName:}" failed. No retries permitted until 2026-07-13 14:54:01.977043314 +0000 UTC m=+68.174196240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-jd8gw" (UID: "5471e5ad-8965-4379-af6a-2cb21fb280e7") : secret "kube-state-metrics-tls" not found
Jul 13 14:54:01.477125 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.477100    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.477125 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.477113    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/579a5f61-d453-4327-ab8d-ea6dae5d44c9-metrics-client-ca\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.477690 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.477181    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/579a5f61-d453-4327-ab8d-ea6dae5d44c9-root\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.477690 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.477273    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5471e5ad-8965-4379-af6a-2cb21fb280e7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.477690 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:54:01.477324    2564 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 13 14:54:01.477690 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:54:01.477358    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8f2fc05-e0b7-4131-b42b-41aa944c7679-openshift-state-metrics-tls podName:e8f2fc05-e0b7-4131-b42b-41aa944c7679 nodeName:}" failed. No retries permitted until 2026-07-13 14:54:01.977346324 +0000 UTC m=+68.174499249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e8f2fc05-e0b7-4131-b42b-41aa944c7679-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-5w66h" (UID: "e8f2fc05-e0b7-4131-b42b-41aa944c7679") : secret "openshift-state-metrics-tls" not found
Jul 13 14:54:01.477690 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.477500    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-textfile\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.477690 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:54:01.477649    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 13 14:54:01.477690 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:54:01.477733    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-tls podName:579a5f61-d453-4327-ab8d-ea6dae5d44c9 nodeName:}" failed. No retries permitted until 2026-07-13 14:54:01.977686356 +0000 UTC m=+68.174839286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-tls") pod "node-exporter-2mnl5" (UID: "579a5f61-d453-4327-ab8d-ea6dae5d44c9") : secret "node-exporter-tls" not found
Jul 13 14:54:01.477690 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.477771    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/579a5f61-d453-4327-ab8d-ea6dae5d44c9-sys\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.478422 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.477960    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.478422 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.477999    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5471e5ad-8965-4379-af6a-2cb21fb280e7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.478422 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.478119    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8f2fc05-e0b7-4131-b42b-41aa944c7679-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5w66h\" (UID: \"e8f2fc05-e0b7-4131-b42b-41aa944c7679\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:01.480586 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.480555    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8f2fc05-e0b7-4131-b42b-41aa944c7679-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5w66h\" (UID: \"e8f2fc05-e0b7-4131-b42b-41aa944c7679\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:01.480798 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.480771    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.482400 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.482350    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.486419 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.486390    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-587c6\" (UniqueName: \"kubernetes.io/projected/e8f2fc05-e0b7-4131-b42b-41aa944c7679-kube-api-access-587c6\") pod \"openshift-state-metrics-84bd68b896-5w66h\" (UID: \"e8f2fc05-e0b7-4131-b42b-41aa944c7679\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:01.486868 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.486824    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftnnj\" (UniqueName: \"kubernetes.io/projected/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-api-access-ftnnj\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.491384 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.491359    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzmz7\" (UniqueName: \"kubernetes.io/projected/579a5f61-d453-4327-ab8d-ea6dae5d44c9-kube-api-access-rzmz7\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.659815 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.659773    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9pghm" event={"ID":"93029547-ab84-4b4f-ac0f-e1b86f5ca5de","Type":"ContainerStarted","Data":"8575d755cb32d133d777c4141ae7d845f04361cf2191fc0dd554319fb5e2ee9d"}
Jul 13 14:54:01.660252 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.659842    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9pghm" event={"ID":"93029547-ab84-4b4f-ac0f-e1b86f5ca5de","Type":"ContainerStarted","Data":"a0bfe70ec7f03d8ab53cb33d8498730e25602420900090cd6a3a1d1b4f161f52"}
Jul 13 14:54:01.682609 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.682551    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9pghm" podStartSLOduration=66.699394613 podStartE2EDuration="1m7.682534547s" totalImagesPullingTime="983.139934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:01.682026591 +0000 UTC m=+67.879179542" watchObservedRunningTime="2026-07-13 14:54:01.682534547 +0000 UTC m=+67.879687494"
Jul 13 14:54:01.981685 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.981650    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-tls\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.981880 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.981731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f2fc05-e0b7-4131-b42b-41aa944c7679-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5w66h\" (UID: \"e8f2fc05-e0b7-4131-b42b-41aa944c7679\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:01.981880 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.981763    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:01.982007 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:54:01.981909    2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 13 14:54:01.982007 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:54:01.981983    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-tls podName:5471e5ad-8965-4379-af6a-2cb21fb280e7 nodeName:}" failed. No retries permitted until 2026-07-13 14:54:02.981963892 +0000 UTC m=+69.179116832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-jd8gw" (UID: "5471e5ad-8965-4379-af6a-2cb21fb280e7") : secret "kube-state-metrics-tls" not found
Jul 13 14:54:01.984836 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.984785    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/579a5f61-d453-4327-ab8d-ea6dae5d44c9-node-exporter-tls\") pod \"node-exporter-2mnl5\" (UID: \"579a5f61-d453-4327-ab8d-ea6dae5d44c9\") " pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:01.984963 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:01.984846    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f2fc05-e0b7-4131-b42b-41aa944c7679-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5w66h\" (UID: \"e8f2fc05-e0b7-4131-b42b-41aa944c7679\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:02.175095 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:02.175064    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"
Jul 13 14:54:02.201765 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:02.201562    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2mnl5"
Jul 13 14:54:02.220688 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:54:02.220647    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579a5f61_d453_4327_ab8d_ea6dae5d44c9.slice/crio-75c6a387972f14a955d3ca954d53a6c6b2a34ab2eedb59fb7919c93e0b2839a4 WatchSource:0}: Error finding container 75c6a387972f14a955d3ca954d53a6c6b2a34ab2eedb59fb7919c93e0b2839a4: Status 404 returned error can't find the container with id 75c6a387972f14a955d3ca954d53a6c6b2a34ab2eedb59fb7919c93e0b2839a4
Jul 13 14:54:02.344181 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:02.344148    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h"]
Jul 13 14:54:02.666006 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:02.665964    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2mnl5" event={"ID":"579a5f61-d453-4327-ab8d-ea6dae5d44c9","Type":"ContainerStarted","Data":"75c6a387972f14a955d3ca954d53a6c6b2a34ab2eedb59fb7919c93e0b2839a4"}
Jul 13 14:54:02.668529 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:02.668481    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h" event={"ID":"e8f2fc05-e0b7-4131-b42b-41aa944c7679","Type":"ContainerStarted","Data":"115889e927c4942c60fba9658f88d7df51e1cb73d51825e6e6c3af19e8fb5dad"}
Jul 13 14:54:02.668663 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:02.668542    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h" event={"ID":"e8f2fc05-e0b7-4131-b42b-41aa944c7679","Type":"ContainerStarted","Data":"4b8df8acea69a65c96f074151450d3c2eb2e1bd095ec598a2bee4cd3fddf8b71"}
Jul 13 14:54:02.668663 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:02.668570    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h" event={"ID":"e8f2fc05-e0b7-4131-b42b-41aa944c7679","Type":"ContainerStarted","Data":"f007660f238815bf1d528fc12b12ae3aa775959e70262d50da5f0cd0f4ff050a"}
Jul 13 14:54:02.991145 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:02.991105    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:02.997465 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:02.997376    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5471e5ad-8965-4379-af6a-2cb21fb280e7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-jd8gw\" (UID: \"5471e5ad-8965-4379-af6a-2cb21fb280e7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:03.087415 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:03.087377    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"
Jul 13 14:54:04.234720 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.234660    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-77bfd56947-67k5j"]
Jul 13 14:54:04.263890 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.263847    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77bfd56947-67k5j"]
Jul 13 14:54:04.264036 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.263943    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.267089 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.266226    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 13 14:54:04.267089 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.266396    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 13 14:54:04.267089 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.266473    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 13 14:54:04.267089 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.266654    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 13 14:54:04.267089 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.266657    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2f8gn\""
Jul 13 14:54:04.267089 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.266903    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cv0er731q3pql\""
Jul 13 14:54:04.267089 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.266920    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 13 14:54:04.301852 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.301821    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.302015 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.301864    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-metrics-client-ca\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.302015 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.301890    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-grpc-tls\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.302015 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.301914    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.302015 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.301941    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.302199 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.302026    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.302199 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.302050    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfvm9\" (UniqueName: \"kubernetes.io/projected/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-kube-api-access-mfvm9\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.302199 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.302075    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-tls\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.402466 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.402429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.402466 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.402470    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfvm9\" (UniqueName: \"kubernetes.io/projected/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-kube-api-access-mfvm9\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.402755 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.402494    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-tls\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.402755 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.402535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.402755 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.402565    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-metrics-client-ca\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.402755 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.402590    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-grpc-tls\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.402755 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.402615    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.402755 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.402645    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.403841 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.403811    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-metrics-client-ca\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.405498 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.405471    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.406113 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.405879    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-tls\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.406113 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.405921    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-grpc-tls\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.406113 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.406073    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.406410 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.406392    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.406563 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.406542    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.411155 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.411137    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfvm9\" (UniqueName: \"kubernetes.io/projected/7d2ac378-5a0a-48d0-8ba1-950196dd7efc-kube-api-access-mfvm9\") pod \"thanos-querier-77bfd56947-67k5j\" (UID: \"7d2ac378-5a0a-48d0-8ba1-950196dd7efc\") " pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:04.574891 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:04.574804    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:05.634427 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:05.634396    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4dd6p"
Jul 13 14:54:05.979111 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:05.979077    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s"]
Jul 13 14:54:05.982277 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:05.982253    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s"
Jul 13 14:54:05.984552 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:05.984530    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 13 14:54:05.984676 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:05.984585    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2v4wg\""
Jul 13 14:54:05.991432 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:05.991374    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s"]
Jul 13 14:54:06.018057 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.018026    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xczw4\" (UniqueName: \"kubernetes.io/projected/9b8aa987-baea-42b4-814c-c92754534afc-kube-api-access-xczw4\") pod \"monitoring-plugin-5f68df7f58-z7g2s\" (UID: \"9b8aa987-baea-42b4-814c-c92754534afc\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s"
Jul 13 14:54:06.018183 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.018080    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b8aa987-baea-42b4-814c-c92754534afc-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-z7g2s\" (UID: \"9b8aa987-baea-42b4-814c-c92754534afc\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s"
Jul 13 14:54:06.121525 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.120979    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xczw4\" (UniqueName: \"kubernetes.io/projected/9b8aa987-baea-42b4-814c-c92754534afc-kube-api-access-xczw4\") pod \"monitoring-plugin-5f68df7f58-z7g2s\" (UID: \"9b8aa987-baea-42b4-814c-c92754534afc\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s"
Jul 13 14:54:06.121525 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.121029    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b8aa987-baea-42b4-814c-c92754534afc-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-z7g2s\" (UID: \"9b8aa987-baea-42b4-814c-c92754534afc\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s"
Jul 13 14:54:06.121525 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:54:06.121206    2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 13 14:54:06.121525 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:54:06.121267    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b8aa987-baea-42b4-814c-c92754534afc-monitoring-plugin-cert podName:9b8aa987-baea-42b4-814c-c92754534afc nodeName:}" failed. No retries permitted until 2026-07-13 14:54:06.621243805 +0000 UTC m=+72.818396733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9b8aa987-baea-42b4-814c-c92754534afc-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-z7g2s" (UID: "9b8aa987-baea-42b4-814c-c92754534afc") : secret "monitoring-plugin-cert" not found
Jul 13 14:54:06.131943 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.131887    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xczw4\" (UniqueName: \"kubernetes.io/projected/9b8aa987-baea-42b4-814c-c92754534afc-kube-api-access-xczw4\") pod \"monitoring-plugin-5f68df7f58-z7g2s\" (UID: \"9b8aa987-baea-42b4-814c-c92754534afc\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s"
Jul 13 14:54:06.161900 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.161605    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-jd8gw"]
Jul 13 14:54:06.166201 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:54:06.166170    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5471e5ad_8965_4379_af6a_2cb21fb280e7.slice/crio-492facf4cb596f0fab531f8747e9dad0715e9e11fa079649a48044a1925643f3 WatchSource:0}: Error finding container 492facf4cb596f0fab531f8747e9dad0715e9e11fa079649a48044a1925643f3: Status 404 returned error can't find the container with id 492facf4cb596f0fab531f8747e9dad0715e9e11fa079649a48044a1925643f3
Jul 13 14:54:06.194807 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.194614    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77bfd56947-67k5j"]
Jul 13 14:54:06.197026 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:54:06.196982    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2ac378_5a0a_48d0_8ba1_950196dd7efc.slice/crio-0798438c68f18d163af2fffd50fd072f15d7d159e3704135aa00f1a079a20c7e WatchSource:0}: Error finding container 0798438c68f18d163af2fffd50fd072f15d7d159e3704135aa00f1a079a20c7e: Status 404 returned error can't find the container with id 0798438c68f18d163af2fffd50fd072f15d7d159e3704135aa00f1a079a20c7e
Jul 13 14:54:06.626943 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.626912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b8aa987-baea-42b4-814c-c92754534afc-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-z7g2s\" (UID: \"9b8aa987-baea-42b4-814c-c92754534afc\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s"
Jul 13 14:54:06.629342 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.629325    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b8aa987-baea-42b4-814c-c92754534afc-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-z7g2s\" (UID: \"9b8aa987-baea-42b4-814c-c92754534afc\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s"
Jul 13 14:54:06.683801 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.683765    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hl7zq" event={"ID":"af9196c9-6c2a-466c-bea6-fd690110ef71","Type":"ContainerStarted","Data":"3b122137fe3aa4a66bac90772c2923b5576aae70df937c7999d500a475e0854f"}
Jul 13 14:54:06.684208 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.683875    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:54:06.685310 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.685287    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jgpsr" event={"ID":"bdef54cb-b0a3-4783-9655-5296a4f95022","Type":"ContainerStarted","Data":"072a3ecc84a239e3fcfac8a20392f721f6e06cf67c9dc455db19c4f09db12246"}
Jul 13 14:54:06.686441 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.686420    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j" event={"ID":"7d2ac378-5a0a-48d0-8ba1-950196dd7efc","Type":"ContainerStarted","Data":"0798438c68f18d163af2fffd50fd072f15d7d159e3704135aa00f1a079a20c7e"}
Jul 13 14:54:06.688021 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.687999    2564 generic.go:356] "Generic (PLEG): container finished" podID="579a5f61-d453-4327-ab8d-ea6dae5d44c9" containerID="0e378ae564612e628b1b6b26308f14b81ef02745d0c246e229cff4e6f72c7040" exitCode=0
Jul 13 14:54:06.688091 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.688078    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2mnl5" event={"ID":"579a5f61-d453-4327-ab8d-ea6dae5d44c9","Type":"ContainerDied","Data":"0e378ae564612e628b1b6b26308f14b81ef02745d0c246e229cff4e6f72c7040"}
Jul 13 14:54:06.692265 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.692231    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h" event={"ID":"e8f2fc05-e0b7-4131-b42b-41aa944c7679","Type":"ContainerStarted","Data":"f49da9e4c8556cdc39801351efcd62221cd82485455ea1ae00ea939b7bdf1518"}
Jul 13 14:54:06.693513 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.693494    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw" event={"ID":"5471e5ad-8965-4379-af6a-2cb21fb280e7","Type":"ContainerStarted","Data":"492facf4cb596f0fab531f8747e9dad0715e9e11fa079649a48044a1925643f3"}
Jul 13 14:54:06.703603 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.703564    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hl7zq" podStartSLOduration=67.191786051 podStartE2EDuration="1m12.703550281s" totalImagesPullingTime="5.51176423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:06.702823974 +0000 UTC m=+72.899976937" watchObservedRunningTime="2026-07-13 14:54:06.703550281 +0000 UTC m=+72.900703262"
Jul 13 14:54:06.719267 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.719228    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jgpsr" podStartSLOduration=66.891918411 podStartE2EDuration="1m12.719214819s" totalImagesPullingTime="5.827296408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:06.718279955 +0000 UTC m=+72.915432904" watchObservedRunningTime="2026-07-13 14:54:06.719214819 +0000 UTC m=+72.916367765"
Jul 13 14:54:06.738144 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.738100    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5w66h" podStartSLOduration=2.196597567 podStartE2EDuration="5.738076126s" totalImagesPullingTime="3.541478559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:06.736753482 +0000 UTC m=+72.933906435" watchObservedRunningTime="2026-07-13 14:54:06.738076126 +0000 UTC m=+72.935229074"
Jul 13 14:54:06.897401 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:06.897377    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s"
Jul 13 14:54:07.036159 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.036123    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s"]
Jul 13 14:54:07.043308 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:54:07.043278    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8aa987_baea_42b4_814c_c92754534afc.slice/crio-162ffca46f3ce336e21ed9a9efea3a095a9d888aa156545631570918769b4dae WatchSource:0}: Error finding container 162ffca46f3ce336e21ed9a9efea3a095a9d888aa156545631570918769b4dae: Status 404 returned error can't find the container with id 162ffca46f3ce336e21ed9a9efea3a095a9d888aa156545631570918769b4dae
Jul 13 14:54:07.492871 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.492829    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 14:54:07.498658 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.497294    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.500274 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.500243    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 13 14:54:07.500561 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.500540    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 13 14:54:07.500915 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.500899    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 13 14:54:07.500994 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.500912    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 13 14:54:07.501894 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.501119    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 13 14:54:07.501894 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.501286    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 13 14:54:07.501894 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.501355    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5ek7jv4ais3bp\""
Jul 13 14:54:07.501894 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.501122    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 13 14:54:07.501894 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.501827    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 13 14:54:07.502196 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.502163    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 13 14:54:07.502790 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.502376    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t86hs\""
Jul 13 14:54:07.502790 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.502623    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 13 14:54:07.504831 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.502381    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 13 14:54:07.504831 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.502957    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 13 14:54:07.504831 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.503139    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 13 14:54:07.504831 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.503193    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 13 14:54:07.504831 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.503959    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 13 14:54:07.513808 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.513788    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 14:54:07.635889 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.635849    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-web-config\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.635889 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.635891    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636135 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.635924    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636135 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.635947    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636135 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636135 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2hsc\" (UniqueName: \"kubernetes.io/projected/8788f3ec-784b-40be-9177-6b1e222ac172-kube-api-access-k2hsc\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636135 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636096    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-config\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636135 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636134    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636466 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636466 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636202    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636466 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636238    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636466 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636266    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8788f3ec-784b-40be-9177-6b1e222ac172-config-out\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636466 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8788f3ec-784b-40be-9177-6b1e222ac172-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636466 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636353    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636466 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636392    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636466 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636421    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636466 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636463    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636920 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636487    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636920 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636506    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.636920 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.636525    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.702458 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.702426    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s" event={"ID":"9b8aa987-baea-42b4-814c-c92754534afc","Type":"ContainerStarted","Data":"162ffca46f3ce336e21ed9a9efea3a095a9d888aa156545631570918769b4dae"}
Jul 13 14:54:07.705648 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.705133    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2mnl5" event={"ID":"579a5f61-d453-4327-ab8d-ea6dae5d44c9","Type":"ContainerStarted","Data":"8066406b8a86a183b9da43be229c76eaa56c6ca7168e9043cd448307c280ae85"}
Jul 13 14:54:07.705648 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.705195    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2mnl5" event={"ID":"579a5f61-d453-4327-ab8d-ea6dae5d44c9","Type":"ContainerStarted","Data":"498ccbae09704aa571a6238e00662b6d7acc9ee2d4cbfc9fb2a7cab9938fdfa2"}
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.737720    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.737755    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.737773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.737793    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.737822    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-web-config\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.737836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.737858    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.737879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.737906    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.737935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hsc\" (UniqueName: \"kubernetes.io/projected/8788f3ec-784b-40be-9177-6b1e222ac172-kube-api-access-k2hsc\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.737957    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-config\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.737974    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.737988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.738004    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.738025    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.738046    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8788f3ec-784b-40be-9177-6b1e222ac172-config-out\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.738240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.738064    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8788f3ec-784b-40be-9177-6b1e222ac172-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.739169 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.738079    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.739169 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.738114    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.739169 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.738140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.747026 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.741504    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.747026 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.746786    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.747280 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.747256    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.749382 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.748014    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.749382 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.748142    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.749382 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.748509    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.749382 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.748771    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.749733 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.749383    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.749733 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.749478    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.750274 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.750245    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.750387 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.750254    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.751105 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.751085    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.758888 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.751407    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8788f3ec-784b-40be-9177-6b1e222ac172-config-out\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.758888 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.752476    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8788f3ec-784b-40be-9177-6b1e222ac172-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.758888 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.752916    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-config\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.759565 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.759522    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-web-config\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.761941 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.761916    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.764174 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.764145    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.764646 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.764336    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hsc\" (UniqueName: \"kubernetes.io/projected/8788f3ec-784b-40be-9177-6b1e222ac172-kube-api-access-k2hsc\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.765021 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.764867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:07.812479 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:07.812401    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:08.051141 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:08.050198    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2mnl5" podStartSLOduration=3.204412917 podStartE2EDuration="7.050171155s" totalImagesPullingTime="3.76492372s" totalInitContainerRuntime="80.834518ms" isStatefulPod=true podCreationTimestamp="2026-07-13 14:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:07.730619658 +0000 UTC m=+73.927772605" watchObservedRunningTime="2026-07-13 14:54:08.050171155 +0000 UTC m=+74.247324104"
Jul 13 14:54:08.051141 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:08.050957    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 14:54:08.695483 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:54:08.695451    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8788f3ec_784b_40be_9177_6b1e222ac172.slice/crio-c6c8cd8fc59dc261574c8b78876d5e17676f6474f80cf1748d0d80158db7293b WatchSource:0}: Error finding container c6c8cd8fc59dc261574c8b78876d5e17676f6474f80cf1748d0d80158db7293b: Status 404 returned error can't find the container with id c6c8cd8fc59dc261574c8b78876d5e17676f6474f80cf1748d0d80158db7293b
Jul 13 14:54:08.712032 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:08.712003    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw" event={"ID":"5471e5ad-8965-4379-af6a-2cb21fb280e7","Type":"ContainerStarted","Data":"0fd7c6b50f29adee657ccae8c05f546c0cde12f1edbd4139c081bfecf97adc1c"}
Jul 13 14:54:08.712391 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:08.712073    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw" event={"ID":"5471e5ad-8965-4379-af6a-2cb21fb280e7","Type":"ContainerStarted","Data":"a570d0957b09cf4dc79f3a5b29d8c12092a0c19dfa8110f97e0e6dea84be9a28"}
Jul 13 14:54:08.712391 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:08.712103    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw" event={"ID":"5471e5ad-8965-4379-af6a-2cb21fb280e7","Type":"ContainerStarted","Data":"0b540049fdd6d949a4d5b63215fe0bdc4544de4a6298c87be3764975d5e89a53"}
Jul 13 14:54:08.713322 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:08.713295    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerStarted","Data":"c6c8cd8fc59dc261574c8b78876d5e17676f6474f80cf1748d0d80158db7293b"}
Jul 13 14:54:08.733462 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:08.733395    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-jd8gw" podStartSLOduration=6.246920115 podStartE2EDuration="7.733374139s" totalImagesPullingTime="1.486454024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:08.731791084 +0000 UTC m=+74.928944032" watchObservedRunningTime="2026-07-13 14:54:08.733374139 +0000 UTC m=+74.930527087"
Jul 13 14:54:09.720791 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:09.720750    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j" event={"ID":"7d2ac378-5a0a-48d0-8ba1-950196dd7efc","Type":"ContainerStarted","Data":"bd0f23683d9c9e6c8e48c548b7726504d45bbdc0575a3458f68cf37710304e05"}
Jul 13 14:54:09.721261 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:09.720864    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j" event={"ID":"7d2ac378-5a0a-48d0-8ba1-950196dd7efc","Type":"ContainerStarted","Data":"8bd22c6affe716a866fdc92b8f540b31520b7865b10a01bb598f7357f8b2efad"}
Jul 13 14:54:09.721261 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:09.720912    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j" event={"ID":"7d2ac378-5a0a-48d0-8ba1-950196dd7efc","Type":"ContainerStarted","Data":"aa4c6fdea7fbc8e2f9cd73e27a43492f00fc411aa65fa1dab106d8641a6d5727"}
Jul 13 14:54:09.722930 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:09.722866    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s" event={"ID":"9b8aa987-baea-42b4-814c-c92754534afc","Type":"ContainerStarted","Data":"bf0cc42ea69659848e7c46dfff1c3a7198b93ca2c293f433eb96cf7cb4b1b9ec"}
Jul 13 14:54:09.745835 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:09.745772    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s" podStartSLOduration=2.693190417 podStartE2EDuration="4.745754101s" totalImagesPullingTime="2.052563684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:09.743019689 +0000 UTC m=+75.940172640" watchObservedRunningTime="2026-07-13 14:54:09.745754101 +0000 UTC m=+75.942907048"
Jul 13 14:54:10.729119 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:10.729078    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j" event={"ID":"7d2ac378-5a0a-48d0-8ba1-950196dd7efc","Type":"ContainerStarted","Data":"63a87493aaf79e33fb5968887290fc109f87e19ab96a473eb86732870caf31e8"}
Jul 13 14:54:10.729580 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:10.729143    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j" event={"ID":"7d2ac378-5a0a-48d0-8ba1-950196dd7efc","Type":"ContainerStarted","Data":"6743edbd5444b8f559d3141f96c9280940304b3dc4ea35c7a7257a1a4b371623"}
Jul 13 14:54:10.729580 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:10.729167    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j" event={"ID":"7d2ac378-5a0a-48d0-8ba1-950196dd7efc","Type":"ContainerStarted","Data":"19d5fcae43633f4b5471b779016f148dad4fadfa0807ec643097f93311df1a92"}
Jul 13 14:54:10.729580 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:10.729255    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:10.730750 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:10.730725    2564 generic.go:356] "Generic (PLEG): container finished" podID="8788f3ec-784b-40be-9177-6b1e222ac172" containerID="7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e" exitCode=0
Jul 13 14:54:10.730841 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:10.730809    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerDied","Data":"7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e"}
Jul 13 14:54:10.731060 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:10.731043    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s"
Jul 13 14:54:10.736615 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:10.736599    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z7g2s"
Jul 13 14:54:10.754084 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:10.754042    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j" podStartSLOduration=2.90151179 podStartE2EDuration="6.754030011s" totalImagesPullingTime="3.852518221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:54:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:10.751742963 +0000 UTC m=+76.948895910" watchObservedRunningTime="2026-07-13 14:54:10.754030011 +0000 UTC m=+76.951182958"
Jul 13 14:54:14.751513 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:14.751484    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerStarted","Data":"27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852"}
Jul 13 14:54:14.751893 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:14.751552    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerStarted","Data":"85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02"}
Jul 13 14:54:14.751893 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:14.751579    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerStarted","Data":"75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7"}
Jul 13 14:54:14.751893 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:14.751607    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerStarted","Data":"4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a"}
Jul 13 14:54:14.751893 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:14.751634    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerStarted","Data":"973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4"}
Jul 13 14:54:14.751893 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:14.751657    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerStarted","Data":"26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde"}
Jul 13 14:54:14.785681 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:14.785638    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.332235726 podStartE2EDuration="7.785622892s" totalImagesPullingTime="4.433395732s" totalInitContainerRuntime="19.991434ms" isStatefulPod=false podCreationTimestamp="2026-07-13 14:54:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:54:14.784403217 +0000 UTC m=+80.981556166" watchObservedRunningTime="2026-07-13 14:54:14.785622892 +0000 UTC m=+80.982775905"
Jul 13 14:54:16.742665 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:16.742634    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77bfd56947-67k5j"
Jul 13 14:54:17.813368 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:17.813338    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:54:37.708241 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:54:37.708207    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hl7zq"
Jul 13 14:55:07.813099 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:07.813062    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:07.833153 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:07.833129    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:07.965874 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:07.965845    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:25.899452 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:25.899414    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 14:55:25.900434 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:25.900378    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="prometheus" containerID="cri-o://26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde" gracePeriod=600
Jul 13 14:55:25.900633 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:25.900564    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="kube-rbac-proxy-thanos" containerID="cri-o://27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852" gracePeriod=600
Jul 13 14:55:25.900774 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:25.900618    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="kube-rbac-proxy-web" containerID="cri-o://75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7" gracePeriod=600
Jul 13 14:55:25.900774 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:25.900662    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="kube-rbac-proxy" containerID="cri-o://85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02" gracePeriod=600
Jul 13 14:55:25.900774 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:25.900734    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="thanos-sidecar" containerID="cri-o://4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a" gracePeriod=600
Jul 13 14:55:25.900948 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:25.900672    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="config-reloader" containerID="cri-o://973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4" gracePeriod=600
Jul 13 14:55:26.022113 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.022090    2564 generic.go:356] "Generic (PLEG): container finished" podID="8788f3ec-784b-40be-9177-6b1e222ac172" containerID="75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7" exitCode=0
Jul 13 14:55:26.022113 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.022112    2564 generic.go:356] "Generic (PLEG): container finished" podID="8788f3ec-784b-40be-9177-6b1e222ac172" containerID="4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a" exitCode=0
Jul 13 14:55:26.022275 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.022120    2564 generic.go:356] "Generic (PLEG): container finished" podID="8788f3ec-784b-40be-9177-6b1e222ac172" containerID="973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4" exitCode=0
Jul 13 14:55:26.022275 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.022169    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerDied","Data":"75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7"}
Jul 13 14:55:26.022275 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.022261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerDied","Data":"4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a"}
Jul 13 14:55:26.022392 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.022304    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerDied","Data":"973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4"}
Jul 13 14:55:26.174312 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.174289    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:26.268514 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268486    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-kube-rbac-proxy\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.268671 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268525    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-1\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.268671 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268544    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8788f3ec-784b-40be-9177-6b1e222ac172-config-out\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.268671 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268572    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-db\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.268671 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268612    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-serving-certs-ca-bundle\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.268671 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268638    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-config\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.268671 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268664    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-2\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.269006 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268688    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-metrics-client-ca\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.269006 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268738    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8788f3ec-784b-40be-9177-6b1e222ac172-tls-assets\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.269006 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268770    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-0\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.269006 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268814    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-web-config\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.269006 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268839    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-kubelet-serving-ca-bundle\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.269006 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268868    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 14:55:26.269006 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268892    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-trusted-ca-bundle\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.269006 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268920    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.269006 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268954    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-metrics-client-certs\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.269006 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.268979    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-thanos-prometheus-http-client-file\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.269467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.269014    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.269467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.269024    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 14:55:26.269467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.269042    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-tls\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.269467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.269078    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2hsc\" (UniqueName: \"kubernetes.io/projected/8788f3ec-784b-40be-9177-6b1e222ac172-kube-api-access-k2hsc\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.269467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.269100    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 14:55:26.269467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.269113    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-grpc-tls\") pod \"8788f3ec-784b-40be-9177-6b1e222ac172\" (UID: \"8788f3ec-784b-40be-9177-6b1e222ac172\") "
Jul 13 14:55:26.269467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.269347    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.269467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.269366    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.269467 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.269379    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.269992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.269503    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 14:55:26.269992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.269580    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 14:55:26.269992 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.269852    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 14:55:26.270166 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.270144    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 14:55:26.270557 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.270524    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 14:55:26.272130 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.272095    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8788f3ec-784b-40be-9177-6b1e222ac172-config-out" (OuterVolumeSpecName: "config-out") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 14:55:26.272130 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.272116    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 14:55:26.272360 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.272207    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 14:55:26.272977 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.272920    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-config" (OuterVolumeSpecName: "config") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 14:55:26.273457 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.273432    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 14:55:26.273457 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.273437    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8788f3ec-784b-40be-9177-6b1e222ac172-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 14:55:26.273656 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.273629    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 14:55:26.273781 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.273664    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 14:55:26.274189 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.274166    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 14:55:26.274659 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.274634    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8788f3ec-784b-40be-9177-6b1e222ac172-kube-api-access-k2hsc" (OuterVolumeSpecName: "kube-api-access-k2hsc") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "kube-api-access-k2hsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 14:55:26.274788 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.274769    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 14:55:26.284765 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.284744    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-web-config" (OuterVolumeSpecName: "web-config") pod "8788f3ec-784b-40be-9177-6b1e222ac172" (UID: "8788f3ec-784b-40be-9177-6b1e222ac172"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 14:55:26.370675 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370648    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-web-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.370818 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370679    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.370818 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370719    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.370818 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370731    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.370818 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370739    2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-metrics-client-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.370818 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370768    2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.370818 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370790    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.370818 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370801    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.370818 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370810    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2hsc\" (UniqueName: \"kubernetes.io/projected/8788f3ec-784b-40be-9177-6b1e222ac172-kube-api-access-k2hsc\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.370818 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370819    2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-grpc-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.371114 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370828    2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-secret-kube-rbac-proxy\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.371114 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370836    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8788f3ec-784b-40be-9177-6b1e222ac172-config-out\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.371114 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370844    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-db\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.371114 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370852    2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8788f3ec-784b-40be-9177-6b1e222ac172-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.371114 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370860    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-configmap-metrics-client-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.371114 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370867    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8788f3ec-784b-40be-9177-6b1e222ac172-tls-assets\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:26.371114 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:26.370875    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8788f3ec-784b-40be-9177-6b1e222ac172-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 14:55:27.029086 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.029058    2564 generic.go:356] "Generic (PLEG): container finished" podID="8788f3ec-784b-40be-9177-6b1e222ac172" containerID="27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852" exitCode=0
Jul 13 14:55:27.029086 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.029079    2564 generic.go:356] "Generic (PLEG): container finished" podID="8788f3ec-784b-40be-9177-6b1e222ac172" containerID="85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02" exitCode=0
Jul 13 14:55:27.029086 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.029088    2564 generic.go:356] "Generic (PLEG): container finished" podID="8788f3ec-784b-40be-9177-6b1e222ac172" containerID="26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde" exitCode=0
Jul 13 14:55:27.029544 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.029134    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.029544 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.029150    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerDied","Data":"27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852"}
Jul 13 14:55:27.029544 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.029229    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerDied","Data":"85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02"}
Jul 13 14:55:27.029544 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.029256    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerDied","Data":"26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde"}
Jul 13 14:55:27.029544 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.029282    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8788f3ec-784b-40be-9177-6b1e222ac172","Type":"ContainerDied","Data":"c6c8cd8fc59dc261574c8b78876d5e17676f6474f80cf1748d0d80158db7293b"}
Jul 13 14:55:27.029544 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.029308    2564 scope.go:117] "RemoveContainer" containerID="27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852"
Jul 13 14:55:27.038309 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.038291    2564 scope.go:117] "RemoveContainer" containerID="85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02"
Jul 13 14:55:27.045224 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.045208    2564 scope.go:117] "RemoveContainer" containerID="75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7"
Jul 13 14:55:27.052298 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.051896    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 14:55:27.053286 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.053269    2564 scope.go:117] "RemoveContainer" containerID="4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a"
Jul 13 14:55:27.059288 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.059267    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 14:55:27.060428 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.060412    2564 scope.go:117] "RemoveContainer" containerID="973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4"
Jul 13 14:55:27.067268 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.067252    2564 scope.go:117] "RemoveContainer" containerID="26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde"
Jul 13 14:55:27.074121 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.074107    2564 scope.go:117] "RemoveContainer" containerID="7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e"
Jul 13 14:55:27.080621 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.080604    2564 scope.go:117] "RemoveContainer" containerID="27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852"
Jul 13 14:55:27.080888 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:55:27.080869    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852\": container with ID starting with 27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852 not found: ID does not exist" containerID="27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852"
Jul 13 14:55:27.080951 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.080896    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852"} err="failed to get container status \"27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852\": rpc error: code = NotFound desc = could not find container \"27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852\": container with ID starting with 27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852 not found: ID does not exist"
Jul 13 14:55:27.080951 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.080925    2564 scope.go:117] "RemoveContainer" containerID="85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02"
Jul 13 14:55:27.081147 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:55:27.081127    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02\": container with ID starting with 85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02 not found: ID does not exist" containerID="85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02"
Jul 13 14:55:27.081187 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.081149    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02"} err="failed to get container status \"85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02\": rpc error: code = NotFound desc = could not find container \"85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02\": container with ID starting with 85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02 not found: ID does not exist"
Jul 13 14:55:27.081187 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.081160    2564 scope.go:117] "RemoveContainer" containerID="75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7"
Jul 13 14:55:27.081365 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:55:27.081350    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7\": container with ID starting with 75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7 not found: ID does not exist" containerID="75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7"
Jul 13 14:55:27.081412 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.081367    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7"} err="failed to get container status \"75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7\": rpc error: code = NotFound desc = could not find container \"75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7\": container with ID starting with 75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7 not found: ID does not exist"
Jul 13 14:55:27.081412 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.081379    2564 scope.go:117] "RemoveContainer" containerID="4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a"
Jul 13 14:55:27.081613 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:55:27.081597    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a\": container with ID starting with 4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a not found: ID does not exist" containerID="4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a"
Jul 13 14:55:27.081655 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.081615    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a"} err="failed to get container status \"4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a\": rpc error: code = NotFound desc = could not find container \"4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a\": container with ID starting with 4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a not found: ID does not exist"
Jul 13 14:55:27.081655 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.081628    2564 scope.go:117] "RemoveContainer" containerID="973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4"
Jul 13 14:55:27.081855 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:55:27.081840    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4\": container with ID starting with 973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4 not found: ID does not exist" containerID="973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4"
Jul 13 14:55:27.081946 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.081857    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4"} err="failed to get container status \"973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4\": rpc error: code = NotFound desc = could not find container \"973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4\": container with ID starting with 973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4 not found: ID does not exist"
Jul 13 14:55:27.081946 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.081869    2564 scope.go:117] "RemoveContainer" containerID="26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde"
Jul 13 14:55:27.082095 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:55:27.082076    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde\": container with ID starting with 26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde not found: ID does not exist" containerID="26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde"
Jul 13 14:55:27.082152 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.082107    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde"} err="failed to get container status \"26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde\": rpc error: code = NotFound desc = could not find container \"26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde\": container with ID starting with 26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde not found: ID does not exist"
Jul 13 14:55:27.082152 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.082132    2564 scope.go:117] "RemoveContainer" containerID="7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e"
Jul 13 14:55:27.082354 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:55:27.082338    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e\": container with ID starting with 7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e not found: ID does not exist" containerID="7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e"
Jul 13 14:55:27.082410 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.082363    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e"} err="failed to get container status \"7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e\": rpc error: code = NotFound desc = could not find container \"7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e\": container with ID starting with 7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e not found: ID does not exist"
Jul 13 14:55:27.082410 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.082383    2564 scope.go:117] "RemoveContainer" containerID="27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852"
Jul 13 14:55:27.082591 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.082573    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852"} err="failed to get container status \"27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852\": rpc error: code = NotFound desc = could not find container \"27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852\": container with ID starting with 27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852 not found: ID does not exist"
Jul 13 14:55:27.082654 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.082592    2564 scope.go:117] "RemoveContainer" containerID="85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02"
Jul 13 14:55:27.082816 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.082798    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02"} err="failed to get container status \"85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02\": rpc error: code = NotFound desc = could not find container \"85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02\": container with ID starting with 85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02 not found: ID does not exist"
Jul 13 14:55:27.082859 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.082818    2564 scope.go:117] "RemoveContainer" containerID="75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7"
Jul 13 14:55:27.083009 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.082994    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7"} err="failed to get container status \"75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7\": rpc error: code = NotFound desc = could not find container \"75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7\": container with ID starting with 75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7 not found: ID does not exist"
Jul 13 14:55:27.083049 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.083009    2564 scope.go:117] "RemoveContainer" containerID="4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a"
Jul 13 14:55:27.083227 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.083211    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a"} err="failed to get container status \"4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a\": rpc error: code = NotFound desc = could not find container \"4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a\": container with ID starting with 4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a not found: ID does not exist"
Jul 13 14:55:27.083293 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.083229    2564 scope.go:117] "RemoveContainer" containerID="973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4"
Jul 13 14:55:27.083428 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.083412    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4"} err="failed to get container status \"973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4\": rpc error: code = NotFound desc = could not find container \"973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4\": container with ID starting with 973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4 not found: ID does not exist"
Jul 13 14:55:27.083496 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.083429    2564 scope.go:117] "RemoveContainer" containerID="26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde"
Jul 13 14:55:27.083634 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.083618    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde"} err="failed to get container status \"26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde\": rpc error: code = NotFound desc = could not find container \"26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde\": container with ID starting with 26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde not found: ID does not exist"
Jul 13 14:55:27.083674 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.083634    2564 scope.go:117] "RemoveContainer" containerID="7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e"
Jul 13 14:55:27.083845 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.083827    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e"} err="failed to get container status \"7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e\": rpc error: code = NotFound desc = could not find container \"7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e\": container with ID starting with 7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e not found: ID does not exist"
Jul 13 14:55:27.083921 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.083846    2564 scope.go:117] "RemoveContainer" containerID="27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852"
Jul 13 14:55:27.084048 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.084031    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852"} err="failed to get container status \"27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852\": rpc error: code = NotFound desc = could not find container \"27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852\": container with ID starting with 27f3b8f73ec70efacecbeba70e9e34d7692ad0798cd8d3d3ff75470f32f17852 not found: ID does not exist"
Jul 13 14:55:27.084105 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.084059    2564 scope.go:117] "RemoveContainer" containerID="85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02"
Jul 13 14:55:27.084268 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.084252    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02"} err="failed to get container status \"85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02\": rpc error: code = NotFound desc = could not find container \"85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02\": container with ID starting with 85d4b46c51f528cf5270d9512ac9111a4d95a8628fd5afa159a0eb8231120d02 not found: ID does not exist"
Jul 13 14:55:27.084316 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.084269    2564 scope.go:117] "RemoveContainer" containerID="75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7"
Jul 13 14:55:27.084454 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.084439    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7"} err="failed to get container status \"75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7\": rpc error: code = NotFound desc = could not find container \"75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7\": container with ID starting with 75a66f019927b4207b4a16ec13ff4fbf7d9b5668395e3855039a2819507454e7 not found: ID does not exist"
Jul 13 14:55:27.084501 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.084455    2564 scope.go:117] "RemoveContainer" containerID="4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a"
Jul 13 14:55:27.084626 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.084610    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a"} err="failed to get container status \"4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a\": rpc error: code = NotFound desc = could not find container \"4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a\": container with ID starting with 4d01c991a08c713137b35d9ffe1bbc408edb3478f7bd330c270b6be98833389a not found: ID does not exist"
Jul 13 14:55:27.084668 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.084626    2564 scope.go:117] "RemoveContainer" containerID="973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4"
Jul 13 14:55:27.084813 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.084797    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4"} err="failed to get container status \"973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4\": rpc error: code = NotFound desc = could not find container \"973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4\": container with ID starting with 973118941f0be1e9bbc3b5752b8bb14b8e2b1a94343aae777bd1cbb21a607bd4 not found: ID does not exist"
Jul 13 14:55:27.084813 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.084812    2564 scope.go:117] "RemoveContainer" containerID="26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde"
Jul 13 14:55:27.084993 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.084978    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde"} err="failed to get container status \"26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde\": rpc error: code = NotFound desc = could not find container \"26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde\": container with ID starting with 26c711065c0c15c668e2a16928365b79c5fe727fc62245536bc5a21902da4dde not found: ID does not exist"
Jul 13 14:55:27.085035 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.084993    2564 scope.go:117] "RemoveContainer" containerID="7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e"
Jul 13 14:55:27.085178 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.085158    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e"} err="failed to get container status \"7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e\": rpc error: code = NotFound desc = could not find container \"7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e\": container with ID starting with 7b80854343603fb379ca4bbb75e3968f9695d785e74a5a239169842b8949ec8e not found: ID does not exist"
Jul 13 14:55:27.089901 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.089879    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 14:55:27.090475 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090457    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="prometheus"
Jul 13 14:55:27.090824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090479    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="prometheus"
Jul 13 14:55:27.090824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090502    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="init-config-reloader"
Jul 13 14:55:27.090824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090510    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="init-config-reloader"
Jul 13 14:55:27.090824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090537    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="config-reloader"
Jul 13 14:55:27.090824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090545    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="config-reloader"
Jul 13 14:55:27.090824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090556    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="kube-rbac-proxy-web"
Jul 13 14:55:27.090824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090565    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="kube-rbac-proxy-web"
Jul 13 14:55:27.090824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090587    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="kube-rbac-proxy-thanos"
Jul 13 14:55:27.090824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090595    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="kube-rbac-proxy-thanos"
Jul 13 14:55:27.090824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090620    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="kube-rbac-proxy"
Jul 13 14:55:27.090824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090629    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="kube-rbac-proxy"
Jul 13 14:55:27.090824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090661    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="thanos-sidecar"
Jul 13 14:55:27.090824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090671    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="thanos-sidecar"
Jul 13 14:55:27.090824 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090822    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="kube-rbac-proxy-web"
Jul 13 14:55:27.091505 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090837    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="kube-rbac-proxy"
Jul 13 14:55:27.091505 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090845    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="kube-rbac-proxy-thanos"
Jul 13 14:55:27.091505 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090865    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="config-reloader"
Jul 13 14:55:27.091505 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090883    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="prometheus"
Jul 13 14:55:27.091505 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.090891    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" containerName="thanos-sidecar"
Jul 13 14:55:27.096161 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.096143    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.099373 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.099351    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 13 14:55:27.099521 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.099503    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 13 14:55:27.099868 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.099847    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 13 14:55:27.099949 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.099876    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 13 14:55:27.099949 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.099895    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 13 14:55:27.100090 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.100064    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 13 14:55:27.100164 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.100105    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 13 14:55:27.100164 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.100115    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 13 14:55:27.100164 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.100158    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t86hs\""
Jul 13 14:55:27.100302 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.100231    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5ek7jv4ais3bp\""
Jul 13 14:55:27.100302 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.100248    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 13 14:55:27.100396 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.100117    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 13 14:55:27.100396 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.100162    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 13 14:55:27.100396 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.100356    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 13 14:55:27.101050 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.101035    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 13 14:55:27.103865 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.103842    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 13 14:55:27.106420 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.106401    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 13 14:55:27.111434 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.111414    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 14:55:27.176137 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176078    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176137 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176103    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176137 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176124    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176293 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176141    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176293 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176184    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176293 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176238    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176293 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176268    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb33220c-5612-423e-85a6-3273a704a8bf-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176293 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176288    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176442 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176318    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176442 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176340    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176442 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176361    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176442 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176442 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176424    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176581 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176461    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176581 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176477    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176581 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176491    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176581 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176513    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb33220c-5612-423e-85a6-3273a704a8bf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176581 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176535    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176581 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176553    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-config\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.176581 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.176568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kns2x\" (UniqueName: \"kubernetes.io/projected/bb33220c-5612-423e-85a6-3273a704a8bf-kube-api-access-kns2x\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.277938 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.277901    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.277938 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.277945    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278106 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.277965    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278141 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb33220c-5612-423e-85a6-3273a704a8bf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278141 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278216 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-config\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278216 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278178    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kns2x\" (UniqueName: \"kubernetes.io/projected/bb33220c-5612-423e-85a6-3273a704a8bf-kube-api-access-kns2x\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278358 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278336    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278431 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278374    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278431 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278527 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278435    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278527 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278527 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278498    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278663 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278532    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb33220c-5612-423e-85a6-3273a704a8bf-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278663 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278558    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278663 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278663 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278613    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278663 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278621    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278663 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278629    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278988 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278666    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278988 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278722    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.278988 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.278734    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.279333 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.279279    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.279800 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.279777    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.279965 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.279948    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.281602 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.281574    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.281722 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.281601    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.282049 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.281905    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.282049 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.281957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.282049 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.281968    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-config\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.282257 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.282098    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.282257 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.282157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb33220c-5612-423e-85a6-3273a704a8bf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.282940 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.282857    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.283169 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.283145    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.283940 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.283913    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb33220c-5612-423e-85a6-3273a704a8bf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.284053 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.284032    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb33220c-5612-423e-85a6-3273a704a8bf-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.284470 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.284446    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.284570 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.284448    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.284570 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.284523    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb33220c-5612-423e-85a6-3273a704a8bf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.286241 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.286219    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kns2x\" (UniqueName: \"kubernetes.io/projected/bb33220c-5612-423e-85a6-3273a704a8bf-kube-api-access-kns2x\") pod \"prometheus-k8s-0\" (UID: \"bb33220c-5612-423e-85a6-3273a704a8bf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.406664 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.406634    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:55:27.544810 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:27.544778    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 14:55:27.547897 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:55:27.547875    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb33220c_5612_423e_85a6_3273a704a8bf.slice/crio-813f04372bc08fc83c73ec56dd620a753ca16f9dee57751e5eeb87dd285c49e1 WatchSource:0}: Error finding container 813f04372bc08fc83c73ec56dd620a753ca16f9dee57751e5eeb87dd285c49e1: Status 404 returned error can't find the container with id 813f04372bc08fc83c73ec56dd620a753ca16f9dee57751e5eeb87dd285c49e1
Jul 13 14:55:28.033945 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:28.033908    2564 generic.go:356] "Generic (PLEG): container finished" podID="bb33220c-5612-423e-85a6-3273a704a8bf" containerID="651ff710c148634774a8562877b03740bf2ae6f9525e5d6675c44c3e3c6bbe7e" exitCode=0
Jul 13 14:55:28.034333 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:28.033997    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb33220c-5612-423e-85a6-3273a704a8bf","Type":"ContainerDied","Data":"651ff710c148634774a8562877b03740bf2ae6f9525e5d6675c44c3e3c6bbe7e"}
Jul 13 14:55:28.034333 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:28.034060    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb33220c-5612-423e-85a6-3273a704a8bf","Type":"ContainerStarted","Data":"813f04372bc08fc83c73ec56dd620a753ca16f9dee57751e5eeb87dd285c49e1"}
Jul 13 14:55:28.320035 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:28.319996    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8788f3ec-784b-40be-9177-6b1e222ac172" path="/var/lib/kubelet/pods/8788f3ec-784b-40be-9177-6b1e222ac172/volumes"
Jul 13 14:55:29.041732 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:29.041686    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb33220c-5612-423e-85a6-3273a704a8bf","Type":"ContainerStarted","Data":"f26d42475b040455e599bdf7500fead31bc715707701c7877e2b834d49277ba1"}
Jul 13 14:55:29.042066 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:29.041764    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb33220c-5612-423e-85a6-3273a704a8bf","Type":"ContainerStarted","Data":"5a370802bc18b696616c97856e489f9cd189df77959b7b22bb4bd16b95a60261"}
Jul 13 14:55:29.042066 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:29.041790    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb33220c-5612-423e-85a6-3273a704a8bf","Type":"ContainerStarted","Data":"ea10a6ecf355b62cad355f08a03eb6f685b5fa72bc1f1706f77859cfc6b6daba"}
Jul 13 14:55:29.042066 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:29.041815    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb33220c-5612-423e-85a6-3273a704a8bf","Type":"ContainerStarted","Data":"7481e10f38ee1bafbf7d6159010d6e7654a29479cbf720fe794c9df447953b47"}
Jul 13 14:55:29.042066 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:29.041839    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb33220c-5612-423e-85a6-3273a704a8bf","Type":"ContainerStarted","Data":"683a559b7b54fa3cc3d797ea407b5c105f120e333a972412810ce219750dd771"}
Jul 13 14:55:29.042066 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:29.041861    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb33220c-5612-423e-85a6-3273a704a8bf","Type":"ContainerStarted","Data":"b6bfa408184dd029e61f45e87f8a4ce2f38cf31e1ec03fa936861f7e6baeef25"}
Jul 13 14:55:29.077826 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:29.076405    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.056555605 podStartE2EDuration="2.076387532s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.831927ms" isStatefulPod=false podCreationTimestamp="2026-07-13 14:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:55:29.074788139 +0000 UTC m=+155.271941088" watchObservedRunningTime="2026-07-13 14:55:29.076387532 +0000 UTC m=+155.273540481"
Jul 13 14:55:32.406948 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:55:32.406917    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:56:27.407856 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:56:27.407816    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:56:27.423784 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:56:27.423758    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:56:28.278936 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:56:28.278907    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 14:57:21.670760 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:21.670664    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2m7bl"]
Jul 13 14:57:21.672674 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:21.672657    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2m7bl"
Jul 13 14:57:21.675214 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:21.675191    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 13 14:57:21.675349 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:21.675220    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 13 14:57:21.676176 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:21.676156    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6jh8x\""
Jul 13 14:57:21.681058 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:21.681037    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2m7bl"]
Jul 13 14:57:21.777779 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:21.777750    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkn7k\" (UniqueName: \"kubernetes.io/projected/3f081c03-dad4-4d64-adcd-6c31d8a1d021-kube-api-access-lkn7k\") pod \"openshift-lws-operator-fd8ccff4c-2m7bl\" (UID: \"3f081c03-dad4-4d64-adcd-6c31d8a1d021\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2m7bl"
Jul 13 14:57:21.777921 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:21.777869    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f081c03-dad4-4d64-adcd-6c31d8a1d021-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2m7bl\" (UID: \"3f081c03-dad4-4d64-adcd-6c31d8a1d021\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2m7bl"
Jul 13 14:57:21.878537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:21.878503    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f081c03-dad4-4d64-adcd-6c31d8a1d021-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2m7bl\" (UID: \"3f081c03-dad4-4d64-adcd-6c31d8a1d021\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2m7bl"
Jul 13 14:57:21.878537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:21.878544    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkn7k\" (UniqueName: \"kubernetes.io/projected/3f081c03-dad4-4d64-adcd-6c31d8a1d021-kube-api-access-lkn7k\") pod \"openshift-lws-operator-fd8ccff4c-2m7bl\" (UID: \"3f081c03-dad4-4d64-adcd-6c31d8a1d021\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2m7bl"
Jul 13 14:57:21.878925 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:21.878904    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f081c03-dad4-4d64-adcd-6c31d8a1d021-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2m7bl\" (UID: \"3f081c03-dad4-4d64-adcd-6c31d8a1d021\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2m7bl"
Jul 13 14:57:21.887610 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:21.887586    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkn7k\" (UniqueName: \"kubernetes.io/projected/3f081c03-dad4-4d64-adcd-6c31d8a1d021-kube-api-access-lkn7k\") pod \"openshift-lws-operator-fd8ccff4c-2m7bl\" (UID: \"3f081c03-dad4-4d64-adcd-6c31d8a1d021\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2m7bl"
Jul 13 14:57:21.990757 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:21.990660    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2m7bl"
Jul 13 14:57:22.116248 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:22.116226    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2m7bl"]
Jul 13 14:57:22.118861 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:57:22.118834    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f081c03_dad4_4d64_adcd_6c31d8a1d021.slice/crio-f90928db4165852c023a5543660923d135d89a79e0cbfbcde29ac07b1753ca3e WatchSource:0}: Error finding container f90928db4165852c023a5543660923d135d89a79e0cbfbcde29ac07b1753ca3e: Status 404 returned error can't find the container with id f90928db4165852c023a5543660923d135d89a79e0cbfbcde29ac07b1753ca3e
Jul 13 14:57:22.447063 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:22.447032    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2m7bl" event={"ID":"3f081c03-dad4-4d64-adcd-6c31d8a1d021","Type":"ContainerStarted","Data":"f90928db4165852c023a5543660923d135d89a79e0cbfbcde29ac07b1753ca3e"}
Jul 13 14:57:25.461931 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:25.461854    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2m7bl" event={"ID":"3f081c03-dad4-4d64-adcd-6c31d8a1d021","Type":"ContainerStarted","Data":"7f4de05a6ecb0e27d9e5b0df220541464f3c67b8bc5f50ad4f40857f92315d6d"}
Jul 13 14:57:25.479965 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:25.479911    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2m7bl" podStartSLOduration=1.452829159 podStartE2EDuration="4.479895974s" totalImagesPullingTime="3.027066815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:57:25.477852547 +0000 UTC m=+271.675005494" watchObservedRunningTime="2026-07-13 14:57:25.479895974 +0000 UTC m=+271.677048921"
Jul 13 14:57:45.456251 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.456217    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"]
Jul 13 14:57:45.459519 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.459498    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.462240 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.462220    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 13 14:57:45.462350 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.462247    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 13 14:57:45.462813 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.462792    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 13 14:57:45.462813 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.462806    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4qfmz\""
Jul 13 14:57:45.463047 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.463031    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 13 14:57:45.477242 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.477215    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"]
Jul 13 14:57:45.578848 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.578815    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-manifests\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.579003 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.578854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-webhook-cert\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.579003 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.578893    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-tmp\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.579003 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.578913    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fllc\" (UniqueName: \"kubernetes.io/projected/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-kube-api-access-2fllc\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.579099 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.579008    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.679374 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.679337    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.679544 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.679381    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-manifests\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.679544 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.679411    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-webhook-cert\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.679544 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.679447    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-tmp\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.679544 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.679472    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fllc\" (UniqueName: \"kubernetes.io/projected/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-kube-api-access-2fllc\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.679862 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.679836    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-manifests\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.681829 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.681805    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-tmp\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.682040 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.682022    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.682104 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.682051    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-webhook-cert\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.696537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.696507    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fllc\" (UniqueName: \"kubernetes.io/projected/1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f-kube-api-access-2fllc\") pod \"opendatahub-operator-controller-manager-67985f7df8-pqjp8\" (UID: \"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f\") " pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.772827 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.772741    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:45.907840 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:45.907818    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"]
Jul 13 14:57:45.910444 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:57:45.910419    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cbaabdc_af73_42ba_8d7f_d1b04a2f4f4f.slice/crio-b1b7cea7fdafd7fc8be6336124ee5af23178a10387d10252f26681a4ad30c813 WatchSource:0}: Error finding container b1b7cea7fdafd7fc8be6336124ee5af23178a10387d10252f26681a4ad30c813: Status 404 returned error can't find the container with id b1b7cea7fdafd7fc8be6336124ee5af23178a10387d10252f26681a4ad30c813
Jul 13 14:57:46.541687 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:46.541654    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8" event={"ID":"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f","Type":"ContainerStarted","Data":"b1b7cea7fdafd7fc8be6336124ee5af23178a10387d10252f26681a4ad30c813"}
Jul 13 14:57:49.555122 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:49.555089    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8" event={"ID":"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f","Type":"ContainerStarted","Data":"d460356c61d439447306f3795ee72c3b2086dde67475bc8ceacac3ff605f2013"}
Jul 13 14:57:50.560461 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:50.560431    2564 generic.go:356] "Generic (PLEG): container finished" podID="1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f" containerID="d460356c61d439447306f3795ee72c3b2086dde67475bc8ceacac3ff605f2013" exitCode=0
Jul 13 14:57:50.560971 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:50.560477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8" event={"ID":"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f","Type":"ContainerDied","Data":"d460356c61d439447306f3795ee72c3b2086dde67475bc8ceacac3ff605f2013"}
Jul 13 14:57:51.566193 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:51.566157    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8" event={"ID":"1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f","Type":"ContainerStarted","Data":"e576b501e7cf7edd2aa16c7e1956d0242a3baef54fb50385fa37c4dd2bb42228"}
Jul 13 14:57:51.566567 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:51.566289    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:57:51.594222 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:51.594172    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8" podStartSLOduration=2.138655601 podStartE2EDuration="6.594157461s" totalImagesPullingTime="3.39536498s" totalInitContainerRuntime="1.06013688s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:57:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:57:51.592022045 +0000 UTC m=+297.789174985" watchObservedRunningTime="2026-07-13 14:57:51.594157461 +0000 UTC m=+297.791310409"
Jul 13 14:57:54.190556 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:54.190530    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 14:57:54.193665 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:54.193639    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 14:57:54.194928 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:57:54.194902    2564 kubelet.go:1648] "Image garbage collection succeeded"
Jul 13 14:58:02.572849 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:02.572818    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67985f7df8-pqjp8"
Jul 13 14:58:03.321963 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.321931    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"]
Jul 13 14:58:03.328602 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.328574    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:03.332826 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.332793    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 13 14:58:03.332960 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.332859    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 13 14:58:03.332960 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.332862    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bfjm7\""
Jul 13 14:58:03.332960 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.332898    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 13 14:58:03.343973 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.343947    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"]
Jul 13 14:58:03.428522 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.428493    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx89g\" (UniqueName: \"kubernetes.io/projected/c944fa94-c547-492d-b187-b033865f6951-kube-api-access-nx89g\") pod \"lws-controller-manager-677c5d8dc-pw7xx\" (UID: \"c944fa94-c547-492d-b187-b033865f6951\") " pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:03.428682 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.428540    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c944fa94-c547-492d-b187-b033865f6951-manager-config\") pod \"lws-controller-manager-677c5d8dc-pw7xx\" (UID: \"c944fa94-c547-492d-b187-b033865f6951\") " pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:03.428682 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.428563    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c944fa94-c547-492d-b187-b033865f6951-cert\") pod \"lws-controller-manager-677c5d8dc-pw7xx\" (UID: \"c944fa94-c547-492d-b187-b033865f6951\") " pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:03.428682 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.428635    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c944fa94-c547-492d-b187-b033865f6951-metrics-cert\") pod \"lws-controller-manager-677c5d8dc-pw7xx\" (UID: \"c944fa94-c547-492d-b187-b033865f6951\") " pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:03.529841 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.529813    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c944fa94-c547-492d-b187-b033865f6951-manager-config\") pod \"lws-controller-manager-677c5d8dc-pw7xx\" (UID: \"c944fa94-c547-492d-b187-b033865f6951\") " pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:03.529841 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.529848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c944fa94-c547-492d-b187-b033865f6951-cert\") pod \"lws-controller-manager-677c5d8dc-pw7xx\" (UID: \"c944fa94-c547-492d-b187-b033865f6951\") " pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:03.530064 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.529896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c944fa94-c547-492d-b187-b033865f6951-metrics-cert\") pod \"lws-controller-manager-677c5d8dc-pw7xx\" (UID: \"c944fa94-c547-492d-b187-b033865f6951\") " pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:03.530117 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.530077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx89g\" (UniqueName: \"kubernetes.io/projected/c944fa94-c547-492d-b187-b033865f6951-kube-api-access-nx89g\") pod \"lws-controller-manager-677c5d8dc-pw7xx\" (UID: \"c944fa94-c547-492d-b187-b033865f6951\") " pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:03.530858 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.530836    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c944fa94-c547-492d-b187-b033865f6951-manager-config\") pod \"lws-controller-manager-677c5d8dc-pw7xx\" (UID: \"c944fa94-c547-492d-b187-b033865f6951\") " pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:03.532553 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.532522    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c944fa94-c547-492d-b187-b033865f6951-metrics-cert\") pod \"lws-controller-manager-677c5d8dc-pw7xx\" (UID: \"c944fa94-c547-492d-b187-b033865f6951\") " pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:03.532927 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.532902    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c944fa94-c547-492d-b187-b033865f6951-cert\") pod \"lws-controller-manager-677c5d8dc-pw7xx\" (UID: \"c944fa94-c547-492d-b187-b033865f6951\") " pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:03.541281 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.541255    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx89g\" (UniqueName: \"kubernetes.io/projected/c944fa94-c547-492d-b187-b033865f6951-kube-api-access-nx89g\") pod \"lws-controller-manager-677c5d8dc-pw7xx\" (UID: \"c944fa94-c547-492d-b187-b033865f6951\") " pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:03.638196 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.638115    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:03.769441 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.769408    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"]
Jul 13 14:58:03.772776 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:58:03.772748    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc944fa94_c547_492d_b187_b033865f6951.slice/crio-5c8a8b1e437bedbbccadcd053cc90b66f77dd0eab36e714fcd1ef9f91b2ea4e5 WatchSource:0}: Error finding container 5c8a8b1e437bedbbccadcd053cc90b66f77dd0eab36e714fcd1ef9f91b2ea4e5: Status 404 returned error can't find the container with id 5c8a8b1e437bedbbccadcd053cc90b66f77dd0eab36e714fcd1ef9f91b2ea4e5
Jul 13 14:58:03.774537 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:03.774522    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 14:58:04.617414 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:04.617381    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx" event={"ID":"c944fa94-c547-492d-b187-b033865f6951","Type":"ContainerStarted","Data":"5c8a8b1e437bedbbccadcd053cc90b66f77dd0eab36e714fcd1ef9f91b2ea4e5"}
Jul 13 14:58:05.699333 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.699301    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn"]
Jul 13 14:58:05.702923 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.702890    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn"
Jul 13 14:58:05.708953 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.708928    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 13 14:58:05.709991 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.709801    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 13 14:58:05.709991 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.709832    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 13 14:58:05.709991 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.709808    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sxls8\""
Jul 13 14:58:05.709991 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.709885    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 13 14:58:05.718860 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.718838    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn"]
Jul 13 14:58:05.751235 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.751205    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eecea63b-388b-4f4c-8db4-368cedb166ac-tls-certs\") pod \"kube-auth-proxy-7cfcb69fbd-mqqsn\" (UID: \"eecea63b-388b-4f4c-8db4-368cedb166ac\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn"
Jul 13 14:58:05.751417 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.751251    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8q9q\" (UniqueName: \"kubernetes.io/projected/eecea63b-388b-4f4c-8db4-368cedb166ac-kube-api-access-j8q9q\") pod \"kube-auth-proxy-7cfcb69fbd-mqqsn\" (UID: \"eecea63b-388b-4f4c-8db4-368cedb166ac\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn"
Jul 13 14:58:05.751417 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.751287    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eecea63b-388b-4f4c-8db4-368cedb166ac-tmp\") pod \"kube-auth-proxy-7cfcb69fbd-mqqsn\" (UID: \"eecea63b-388b-4f4c-8db4-368cedb166ac\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn"
Jul 13 14:58:05.852355 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.852325    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8q9q\" (UniqueName: \"kubernetes.io/projected/eecea63b-388b-4f4c-8db4-368cedb166ac-kube-api-access-j8q9q\") pod \"kube-auth-proxy-7cfcb69fbd-mqqsn\" (UID: \"eecea63b-388b-4f4c-8db4-368cedb166ac\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn"
Jul 13 14:58:05.852502 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.852369    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eecea63b-388b-4f4c-8db4-368cedb166ac-tmp\") pod \"kube-auth-proxy-7cfcb69fbd-mqqsn\" (UID: \"eecea63b-388b-4f4c-8db4-368cedb166ac\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn"
Jul 13 14:58:05.852502 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.852434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eecea63b-388b-4f4c-8db4-368cedb166ac-tls-certs\") pod \"kube-auth-proxy-7cfcb69fbd-mqqsn\" (UID: \"eecea63b-388b-4f4c-8db4-368cedb166ac\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn"
Jul 13 14:58:05.854819 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.854796    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eecea63b-388b-4f4c-8db4-368cedb166ac-tmp\") pod \"kube-auth-proxy-7cfcb69fbd-mqqsn\" (UID: \"eecea63b-388b-4f4c-8db4-368cedb166ac\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn"
Jul 13 14:58:05.854946 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.854928    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eecea63b-388b-4f4c-8db4-368cedb166ac-tls-certs\") pod \"kube-auth-proxy-7cfcb69fbd-mqqsn\" (UID: \"eecea63b-388b-4f4c-8db4-368cedb166ac\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn"
Jul 13 14:58:05.865631 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:05.865611    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8q9q\" (UniqueName: \"kubernetes.io/projected/eecea63b-388b-4f4c-8db4-368cedb166ac-kube-api-access-j8q9q\") pod \"kube-auth-proxy-7cfcb69fbd-mqqsn\" (UID: \"eecea63b-388b-4f4c-8db4-368cedb166ac\") " pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn"
Jul 13 14:58:06.014633 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:06.014604    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn"
Jul 13 14:58:06.158457 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:06.158438    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn"]
Jul 13 14:58:06.161544 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:58:06.161517    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeecea63b_388b_4f4c_8db4_368cedb166ac.slice/crio-aa07ffc116c546019e5e24d9298dfc2de44e3d4b43b537b9e2985c59a2e9722a WatchSource:0}: Error finding container aa07ffc116c546019e5e24d9298dfc2de44e3d4b43b537b9e2985c59a2e9722a: Status 404 returned error can't find the container with id aa07ffc116c546019e5e24d9298dfc2de44e3d4b43b537b9e2985c59a2e9722a
Jul 13 14:58:06.626805 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:06.626769    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx" event={"ID":"c944fa94-c547-492d-b187-b033865f6951","Type":"ContainerStarted","Data":"cdc17e9f09d3aae05a3e76558be2967b4decdeebab75da9cb27cb3e9b4f73145"}
Jul 13 14:58:06.626989 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:06.626838    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:06.627998 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:06.627977    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn" event={"ID":"eecea63b-388b-4f4c-8db4-368cedb166ac","Type":"ContainerStarted","Data":"aa07ffc116c546019e5e24d9298dfc2de44e3d4b43b537b9e2985c59a2e9722a"}
Jul 13 14:58:06.649128 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:06.649079    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx" podStartSLOduration=1.509293659 podStartE2EDuration="3.649064315s" totalImagesPullingTime="2.139770656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:58:06.647751849 +0000 UTC m=+312.844904797" watchObservedRunningTime="2026-07-13 14:58:06.649064315 +0000 UTC m=+312.846217265"
Jul 13 14:58:09.642449 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:09.642411    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn" event={"ID":"eecea63b-388b-4f4c-8db4-368cedb166ac","Type":"ContainerStarted","Data":"353cda402b1a59d0f35432df94824a4aa86da00365df88ecec9938e88b9fe252"}
Jul 13 14:58:09.661174 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:09.661124    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cfcb69fbd-mqqsn" podStartSLOduration=1.980571604 podStartE2EDuration="4.661107282s" totalImagesPullingTime="2.680535678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:58:09.660101643 +0000 UTC m=+315.857254592" watchObservedRunningTime="2026-07-13 14:58:09.661107282 +0000 UTC m=+315.858260229"
Jul 13 14:58:17.634434 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:17.634402    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-677c5d8dc-pw7xx"
Jul 13 14:58:17.874984 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:17.874947    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-khsgl"]
Jul 13 14:58:17.878285 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:17.878263    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-khsgl"
Jul 13 14:58:17.881511 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:17.881482    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-5rtrw\""
Jul 13 14:58:17.881787 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:17.881768    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 13 14:58:17.908197 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:17.908164    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-khsgl"]
Jul 13 14:58:17.961839 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:17.961796    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fvkx\" (UniqueName: \"kubernetes.io/projected/f4d9eaad-dda7-4a58-96e9-e7bafc50e442-kube-api-access-8fvkx\") pod \"kserve-controller-manager-d6f4b65b8-khsgl\" (UID: \"f4d9eaad-dda7-4a58-96e9-e7bafc50e442\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-khsgl"
Jul 13 14:58:17.962097 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:17.962078    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4d9eaad-dda7-4a58-96e9-e7bafc50e442-cert\") pod \"kserve-controller-manager-d6f4b65b8-khsgl\" (UID: \"f4d9eaad-dda7-4a58-96e9-e7bafc50e442\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-khsgl"
Jul 13 14:58:18.063681 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:18.063645    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fvkx\" (UniqueName: \"kubernetes.io/projected/f4d9eaad-dda7-4a58-96e9-e7bafc50e442-kube-api-access-8fvkx\") pod \"kserve-controller-manager-d6f4b65b8-khsgl\" (UID: \"f4d9eaad-dda7-4a58-96e9-e7bafc50e442\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-khsgl"
Jul 13 14:58:18.063890 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:18.063831    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4d9eaad-dda7-4a58-96e9-e7bafc50e442-cert\") pod \"kserve-controller-manager-d6f4b65b8-khsgl\" (UID: \"f4d9eaad-dda7-4a58-96e9-e7bafc50e442\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-khsgl"
Jul 13 14:58:18.066485 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:18.066454    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4d9eaad-dda7-4a58-96e9-e7bafc50e442-cert\") pod \"kserve-controller-manager-d6f4b65b8-khsgl\" (UID: \"f4d9eaad-dda7-4a58-96e9-e7bafc50e442\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-khsgl"
Jul 13 14:58:18.077108 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:18.077079    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fvkx\" (UniqueName: \"kubernetes.io/projected/f4d9eaad-dda7-4a58-96e9-e7bafc50e442-kube-api-access-8fvkx\") pod \"kserve-controller-manager-d6f4b65b8-khsgl\" (UID: \"f4d9eaad-dda7-4a58-96e9-e7bafc50e442\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-khsgl"
Jul 13 14:58:18.190543 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:18.190500    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-khsgl"
Jul 13 14:58:18.342425 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:18.342392    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-khsgl"]
Jul 13 14:58:18.345801 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:58:18.345772    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d9eaad_dda7_4a58_96e9_e7bafc50e442.slice/crio-1dfedee58a5189cb3cba13d369972c123b05a9c90688f4281e6a54a96d3cdf5f WatchSource:0}: Error finding container 1dfedee58a5189cb3cba13d369972c123b05a9c90688f4281e6a54a96d3cdf5f: Status 404 returned error can't find the container with id 1dfedee58a5189cb3cba13d369972c123b05a9c90688f4281e6a54a96d3cdf5f
Jul 13 14:58:18.681691 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:18.681654    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-khsgl" event={"ID":"f4d9eaad-dda7-4a58-96e9-e7bafc50e442","Type":"ContainerStarted","Data":"1dfedee58a5189cb3cba13d369972c123b05a9c90688f4281e6a54a96d3cdf5f"}
Jul 13 14:58:21.703776 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:21.703739    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-khsgl" event={"ID":"f4d9eaad-dda7-4a58-96e9-e7bafc50e442","Type":"ContainerStarted","Data":"b12912ae43550ba47de76224db9968a6f3a844e11ec298aff46a3d8e611498d4"}
Jul 13 14:58:21.704211 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:21.703870    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-khsgl"
Jul 13 14:58:21.734907 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:21.734770    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-khsgl" podStartSLOduration=1.921785692 podStartE2EDuration="4.734751581s" totalImagesPullingTime="2.812965889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:58:21.733788071 +0000 UTC m=+327.930941019" watchObservedRunningTime="2026-07-13 14:58:21.734751581 +0000 UTC m=+327.931904532"
Jul 13 14:58:52.713482 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:58:52.713406    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-khsgl"
Jul 13 14:59:55.878550 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:55.878515    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"]
Jul 13 14:59:55.881602 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:55.881582    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"
Jul 13 14:59:55.885379 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:55.885357    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 13 14:59:55.885491 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:55.885362    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 13 14:59:55.885491 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:55.885445    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 13 14:59:55.886121 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:55.886108    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 13 14:59:55.886179 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:55.886168    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ndlxh\""
Jul 13 14:59:55.891832 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:55.891810    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"]
Jul 13 14:59:55.906452 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:55.906424    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/caa479ca-9183-4c1f-8b5e-ed55fc4e8822-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lfwwx\" (UID: \"caa479ca-9183-4c1f-8b5e-ed55fc4e8822\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"
Jul 13 14:59:55.906588 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:55.906472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4txmd\" (UniqueName: \"kubernetes.io/projected/caa479ca-9183-4c1f-8b5e-ed55fc4e8822-kube-api-access-4txmd\") pod \"kuadrant-console-plugin-66b5774985-lfwwx\" (UID: \"caa479ca-9183-4c1f-8b5e-ed55fc4e8822\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"
Jul 13 14:59:55.906588 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:55.906497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/caa479ca-9183-4c1f-8b5e-ed55fc4e8822-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lfwwx\" (UID: \"caa479ca-9183-4c1f-8b5e-ed55fc4e8822\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"
Jul 13 14:59:56.007460 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:56.007424    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4txmd\" (UniqueName: \"kubernetes.io/projected/caa479ca-9183-4c1f-8b5e-ed55fc4e8822-kube-api-access-4txmd\") pod \"kuadrant-console-plugin-66b5774985-lfwwx\" (UID: \"caa479ca-9183-4c1f-8b5e-ed55fc4e8822\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"
Jul 13 14:59:56.007460 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:56.007465    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/caa479ca-9183-4c1f-8b5e-ed55fc4e8822-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lfwwx\" (UID: \"caa479ca-9183-4c1f-8b5e-ed55fc4e8822\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"
Jul 13 14:59:56.007748 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:56.007512    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/caa479ca-9183-4c1f-8b5e-ed55fc4e8822-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lfwwx\" (UID: \"caa479ca-9183-4c1f-8b5e-ed55fc4e8822\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"
Jul 13 14:59:56.007748 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:59:56.007600    2564 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 13 14:59:56.007748 ip-10-0-128-29 kubenswrapper[2564]: E0713 14:59:56.007661    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa479ca-9183-4c1f-8b5e-ed55fc4e8822-plugin-serving-cert podName:caa479ca-9183-4c1f-8b5e-ed55fc4e8822 nodeName:}" failed. No retries permitted until 2026-07-13 14:59:56.507646165 +0000 UTC m=+422.704799091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/caa479ca-9183-4c1f-8b5e-ed55fc4e8822-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-lfwwx" (UID: "caa479ca-9183-4c1f-8b5e-ed55fc4e8822") : secret "plugin-serving-cert" not found
Jul 13 14:59:56.008165 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:56.008147    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/caa479ca-9183-4c1f-8b5e-ed55fc4e8822-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lfwwx\" (UID: \"caa479ca-9183-4c1f-8b5e-ed55fc4e8822\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"
Jul 13 14:59:56.016802 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:56.016777    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4txmd\" (UniqueName: \"kubernetes.io/projected/caa479ca-9183-4c1f-8b5e-ed55fc4e8822-kube-api-access-4txmd\") pod \"kuadrant-console-plugin-66b5774985-lfwwx\" (UID: \"caa479ca-9183-4c1f-8b5e-ed55fc4e8822\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"
Jul 13 14:59:56.512908 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:56.512865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/caa479ca-9183-4c1f-8b5e-ed55fc4e8822-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lfwwx\" (UID: \"caa479ca-9183-4c1f-8b5e-ed55fc4e8822\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"
Jul 13 14:59:56.515425 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:56.515392    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/caa479ca-9183-4c1f-8b5e-ed55fc4e8822-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lfwwx\" (UID: \"caa479ca-9183-4c1f-8b5e-ed55fc4e8822\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"
Jul 13 14:59:56.791205 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:56.791118    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"
Jul 13 14:59:56.920397 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:56.920235    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx"]
Jul 13 14:59:56.923052 ip-10-0-128-29 kubenswrapper[2564]: W0713 14:59:56.923025    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcaa479ca_9183_4c1f_8b5e_ed55fc4e8822.slice/crio-594f87fdbec25f79c3cf24d64f1dc9b94354b41ac90c97ca3395275fdf6e3ae3 WatchSource:0}: Error finding container 594f87fdbec25f79c3cf24d64f1dc9b94354b41ac90c97ca3395275fdf6e3ae3: Status 404 returned error can't find the container with id 594f87fdbec25f79c3cf24d64f1dc9b94354b41ac90c97ca3395275fdf6e3ae3
Jul 13 14:59:57.105684 ip-10-0-128-29 kubenswrapper[2564]: I0713 14:59:57.105600    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx" event={"ID":"caa479ca-9183-4c1f-8b5e-ed55fc4e8822","Type":"ContainerStarted","Data":"594f87fdbec25f79c3cf24d64f1dc9b94354b41ac90c97ca3395275fdf6e3ae3"}
Jul 13 15:00:06.680232 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:06.680191    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fpwvv"]
Jul 13 15:00:06.691107 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:06.691074    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fpwvv"
Jul 13 15:00:06.693753 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:06.693722    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-smh8p\""
Jul 13 15:00:06.694363 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:06.694334    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fpwvv"]
Jul 13 15:00:06.810074 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:06.810038    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbbww\" (UniqueName: \"kubernetes.io/projected/49e17671-ccfe-4d04-8f3a-9d979030c676-kube-api-access-wbbww\") pod \"limitador-operator-controller-manager-859d7b78bd-fpwvv\" (UID: \"49e17671-ccfe-4d04-8f3a-9d979030c676\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fpwvv"
Jul 13 15:00:06.910621 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:06.910580    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbbww\" (UniqueName: \"kubernetes.io/projected/49e17671-ccfe-4d04-8f3a-9d979030c676-kube-api-access-wbbww\") pod \"limitador-operator-controller-manager-859d7b78bd-fpwvv\" (UID: \"49e17671-ccfe-4d04-8f3a-9d979030c676\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fpwvv"
Jul 13 15:00:06.931311 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:06.931231    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbbww\" (UniqueName: \"kubernetes.io/projected/49e17671-ccfe-4d04-8f3a-9d979030c676-kube-api-access-wbbww\") pod \"limitador-operator-controller-manager-859d7b78bd-fpwvv\" (UID: \"49e17671-ccfe-4d04-8f3a-9d979030c676\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fpwvv"
Jul 13 15:00:07.005585 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:07.005554    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fpwvv"
Jul 13 15:00:20.290329 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:20.290221    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fpwvv"]
Jul 13 15:00:20.293000 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:00:20.292970    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e17671_ccfe_4d04_8f3a_9d979030c676.slice/crio-b2a59fff6ea1d9e7005c7fbaf4b9030819b838c4a6f9e0c9bb1e726414aa9660 WatchSource:0}: Error finding container b2a59fff6ea1d9e7005c7fbaf4b9030819b838c4a6f9e0c9bb1e726414aa9660: Status 404 returned error can't find the container with id b2a59fff6ea1d9e7005c7fbaf4b9030819b838c4a6f9e0c9bb1e726414aa9660
Jul 13 15:00:21.231611 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:21.231574    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fpwvv" event={"ID":"49e17671-ccfe-4d04-8f3a-9d979030c676","Type":"ContainerStarted","Data":"b2a59fff6ea1d9e7005c7fbaf4b9030819b838c4a6f9e0c9bb1e726414aa9660"}
Jul 13 15:00:21.233151 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:21.233125    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx" event={"ID":"caa479ca-9183-4c1f-8b5e-ed55fc4e8822","Type":"ContainerStarted","Data":"50fb7f6f1eb8bb3e232af9f03abf48e5d651d87b04b2b9f64001197f96689ee4"}
Jul 13 15:00:21.254126 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:21.254074    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfwwx" podStartSLOduration=2.62350631 podStartE2EDuration="26.254056241s" totalImagesPullingTime="23.630549931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:00:21.252424374 +0000 UTC m=+447.449577322" watchObservedRunningTime="2026-07-13 15:00:21.254056241 +0000 UTC m=+447.451209229"
Jul 13 15:00:25.255301 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:25.255265    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fpwvv" event={"ID":"49e17671-ccfe-4d04-8f3a-9d979030c676","Type":"ContainerStarted","Data":"b1b644d2e7254001042ed0ca50c4a9bff2b165430f3e3b2c4a9956d0b97a1d00"}
Jul 13 15:00:25.255722 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:25.255352    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fpwvv"
Jul 13 15:00:25.273762 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:25.273714    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fpwvv" podStartSLOduration=15.369512267 podStartE2EDuration="19.273679972s" totalImagesPullingTime="3.904167705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:00:25.272108747 +0000 UTC m=+451.469261695" watchObservedRunningTime="2026-07-13 15:00:25.273679972 +0000 UTC m=+451.470832919"
Jul 13 15:00:36.262623 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:36.262587    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fpwvv"
Jul 13 15:00:51.297006 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:51.296971    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-zfdq6"]
Jul 13 15:00:51.377335 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:51.377307    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zfdq6"]
Jul 13 15:00:51.377499 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:51.377400    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zfdq6"
Jul 13 15:00:51.380820 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:51.380800    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xhtrg\""
Jul 13 15:00:51.504913 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:51.504877    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js4pg\" (UniqueName: \"kubernetes.io/projected/688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef-kube-api-access-js4pg\") pod \"authorino-75d556bd77-zfdq6\" (UID: \"688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef\") " pod="kuadrant-system/authorino-75d556bd77-zfdq6"
Jul 13 15:00:51.605600 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:51.605503    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js4pg\" (UniqueName: \"kubernetes.io/projected/688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef-kube-api-access-js4pg\") pod \"authorino-75d556bd77-zfdq6\" (UID: \"688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef\") " pod="kuadrant-system/authorino-75d556bd77-zfdq6"
Jul 13 15:00:51.614139 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:51.614110    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js4pg\" (UniqueName: \"kubernetes.io/projected/688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef-kube-api-access-js4pg\") pod \"authorino-75d556bd77-zfdq6\" (UID: \"688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef\") " pod="kuadrant-system/authorino-75d556bd77-zfdq6"
Jul 13 15:00:51.686626 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:51.686594    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zfdq6"
Jul 13 15:00:51.822690 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:51.822647    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zfdq6"]
Jul 13 15:00:51.826883 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:00:51.826853    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod688ea2cd_2ea9_4ce5_ad1b_5229f51c91ef.slice/crio-0169f9a45b2a7e17f42f190b9890779ad67b62a093155e297f217f31ae8decd9 WatchSource:0}: Error finding container 0169f9a45b2a7e17f42f190b9890779ad67b62a093155e297f217f31ae8decd9: Status 404 returned error can't find the container with id 0169f9a45b2a7e17f42f190b9890779ad67b62a093155e297f217f31ae8decd9
Jul 13 15:00:52.372439 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:52.372398    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zfdq6" event={"ID":"688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef","Type":"ContainerStarted","Data":"0169f9a45b2a7e17f42f190b9890779ad67b62a093155e297f217f31ae8decd9"}
Jul 13 15:00:58.401952 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:58.401907    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zfdq6" event={"ID":"688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef","Type":"ContainerStarted","Data":"929aead1b7a0538c80d897dcc33080f7f32f7f0a42bae195e2190b6f9fb8d374"}
Jul 13 15:00:58.420510 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:00:58.420459    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-zfdq6" podStartSLOduration=1.561590411 podStartE2EDuration="7.420443258s" totalImagesPullingTime="5.858852847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:00:58.419499477 +0000 UTC m=+484.616652426" watchObservedRunningTime="2026-07-13 15:00:58.420443258 +0000 UTC m=+484.617596206"
Jul 13 15:01:35.413941 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:01:35.413898    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zfdq6"]
Jul 13 15:01:35.414486 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:01:35.414113    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-zfdq6" podUID="688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef" containerName="authorino" containerID="cri-o://929aead1b7a0538c80d897dcc33080f7f32f7f0a42bae195e2190b6f9fb8d374" gracePeriod=30
Jul 13 15:01:35.566876 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:01:35.566837    2564 generic.go:356] "Generic (PLEG): container finished" podID="688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef" containerID="929aead1b7a0538c80d897dcc33080f7f32f7f0a42bae195e2190b6f9fb8d374" exitCode=0
Jul 13 15:01:35.567057 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:01:35.566899    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zfdq6" event={"ID":"688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef","Type":"ContainerDied","Data":"929aead1b7a0538c80d897dcc33080f7f32f7f0a42bae195e2190b6f9fb8d374"}
Jul 13 15:01:35.660910 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:01:35.660882    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zfdq6"
Jul 13 15:01:35.695259 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:01:35.695179    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js4pg\" (UniqueName: \"kubernetes.io/projected/688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef-kube-api-access-js4pg\") pod \"688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef\" (UID: \"688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef\") "
Jul 13 15:01:35.697486 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:01:35.697463    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef-kube-api-access-js4pg" (OuterVolumeSpecName: "kube-api-access-js4pg") pod "688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef" (UID: "688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef"). InnerVolumeSpecName "kube-api-access-js4pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:01:35.796102 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:01:35.796060    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js4pg\" (UniqueName: \"kubernetes.io/projected/688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef-kube-api-access-js4pg\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:01:36.572545 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:01:36.572513    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zfdq6" event={"ID":"688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef","Type":"ContainerDied","Data":"0169f9a45b2a7e17f42f190b9890779ad67b62a093155e297f217f31ae8decd9"}
Jul 13 15:01:36.572545 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:01:36.572542    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zfdq6"
Jul 13 15:01:36.573024 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:01:36.572557    2564 scope.go:117] "RemoveContainer" containerID="929aead1b7a0538c80d897dcc33080f7f32f7f0a42bae195e2190b6f9fb8d374"
Jul 13 15:01:36.589482 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:01:36.589452    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zfdq6"]
Jul 13 15:01:36.592797 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:01:36.592775    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zfdq6"]
Jul 13 15:01:38.316773 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:01:38.316739    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef" path="/var/lib/kubelet/pods/688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef/volumes"
Jul 13 15:02:42.672640 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.672604    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"]
Jul 13 15:02:42.673114 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.672946    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef" containerName="authorino"
Jul 13 15:02:42.673114 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.672957    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef" containerName="authorino"
Jul 13 15:02:42.673114 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.673026    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="688ea2cd-2ea9-4ce5-ad1b-5229f51c91ef" containerName="authorino"
Jul 13 15:02:42.674895 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.674880    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.678186 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.678164    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 13 15:02:42.678319 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.678165    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ttcck\""
Jul 13 15:02:42.678319 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.678247    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 13 15:02:42.678319 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.678261    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 13 15:02:42.685263 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.685240    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"]
Jul 13 15:02:42.785299 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.785255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.785474 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.785310    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.785474 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.785336    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.785474 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.785397    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6rx5\" (UniqueName: \"kubernetes.io/projected/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-kube-api-access-c6rx5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.785474 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.785455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.785650 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.785490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.886939 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.886904    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.887129 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.886966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.887129 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.886998    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.887129 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.887025    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.887129 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.887074    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rx5\" (UniqueName: \"kubernetes.io/projected/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-kube-api-access-c6rx5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.887320 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.887134    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.887381 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.887342    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.887435 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.887397    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.887514 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.887496    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.889431 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.889409    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.889592 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.889575    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.896599 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.896579    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6rx5\" (UniqueName: \"kubernetes.io/projected/fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b-kube-api-access-c6rx5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6\" (UID: \"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:42.985499 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:42.985412    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:43.122205 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:43.122176    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"]
Jul 13 15:02:43.126192 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:02:43.126164    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb2ffcb_5103_4e3e_b3e9_debacba0fc7b.slice/crio-b48108537cdfbdca2e55944a5049ddda1ae8cc8a46a00d260f66e9c0563c10c8 WatchSource:0}: Error finding container b48108537cdfbdca2e55944a5049ddda1ae8cc8a46a00d260f66e9c0563c10c8: Status 404 returned error can't find the container with id b48108537cdfbdca2e55944a5049ddda1ae8cc8a46a00d260f66e9c0563c10c8
Jul 13 15:02:43.871563 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:43.871514    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6" event={"ID":"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b","Type":"ContainerStarted","Data":"b48108537cdfbdca2e55944a5049ddda1ae8cc8a46a00d260f66e9c0563c10c8"}
Jul 13 15:02:44.877815 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:44.877774    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6" event={"ID":"fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b","Type":"ContainerStarted","Data":"779041f5dbb8d5b43d20b33c31307081113685a98aa69e443684bd4fe91d9445"}
Jul 13 15:02:44.896949 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:44.896899    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6" podStartSLOduration=1.81005137 podStartE2EDuration="2.896881962s" totalImagesPullingTime="1.086830592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:02:44.895851491 +0000 UTC m=+591.093004439" watchObservedRunningTime="2026-07-13 15:02:44.896881962 +0000 UTC m=+591.094034910"
Jul 13 15:02:52.986230 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:52.986188    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:52.986799 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:52.986249    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:52.998877 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:52.998850    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:53.929473 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:53.929447    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6"
Jul 13 15:02:54.219141 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:54.219064    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:02:54.221947 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:02:54.221925    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:03:08.975089 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:08.975049    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"]
Jul 13 15:03:08.977727 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:08.977686    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:08.980962 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:08.980936    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 13 15:03:08.989836 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:08.989809    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"]
Jul 13 15:03:09.010865 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.010828    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc7z4\" (UniqueName: \"kubernetes.io/projected/c6765fa4-2372-4092-bc64-2601b628519a-kube-api-access-bc7z4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.011063 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.010883    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6765fa4-2372-4092-bc64-2601b628519a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.011063 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.010903    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6765fa4-2372-4092-bc64-2601b628519a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.011063 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.011023    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6765fa4-2372-4092-bc64-2601b628519a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.011172 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.011097    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6765fa4-2372-4092-bc64-2601b628519a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.011172 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.011126    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6765fa4-2372-4092-bc64-2601b628519a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.111885 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.111844    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc7z4\" (UniqueName: \"kubernetes.io/projected/c6765fa4-2372-4092-bc64-2601b628519a-kube-api-access-bc7z4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.112088 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.111905    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6765fa4-2372-4092-bc64-2601b628519a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.112088 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.111933    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6765fa4-2372-4092-bc64-2601b628519a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.112088 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.111967    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6765fa4-2372-4092-bc64-2601b628519a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.112241 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.112115    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6765fa4-2372-4092-bc64-2601b628519a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.112241 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.112164    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6765fa4-2372-4092-bc64-2601b628519a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.112389 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.112363    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6765fa4-2372-4092-bc64-2601b628519a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.112447 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.112390    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6765fa4-2372-4092-bc64-2601b628519a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.112548 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.112529    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6765fa4-2372-4092-bc64-2601b628519a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.114404 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.114381    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6765fa4-2372-4092-bc64-2601b628519a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.114775 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.114753    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6765fa4-2372-4092-bc64-2601b628519a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.120800 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.120773    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc7z4\" (UniqueName: \"kubernetes.io/projected/c6765fa4-2372-4092-bc64-2601b628519a-kube-api-access-bc7z4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p\" (UID: \"c6765fa4-2372-4092-bc64-2601b628519a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.296974 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.296872    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:09.438466 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.438441    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"]
Jul 13 15:03:09.441197 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:03:09.441163    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6765fa4_2372_4092_bc64_2601b628519a.slice/crio-86f4550e96aa4dd6c77ef12da92b966900568b0621682211b229e8663bbcd41c WatchSource:0}: Error finding container 86f4550e96aa4dd6c77ef12da92b966900568b0621682211b229e8663bbcd41c: Status 404 returned error can't find the container with id 86f4550e96aa4dd6c77ef12da92b966900568b0621682211b229e8663bbcd41c
Jul 13 15:03:09.442609 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:09.442594    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 15:03:10.001367 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:10.001277    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p" event={"ID":"c6765fa4-2372-4092-bc64-2601b628519a","Type":"ContainerStarted","Data":"0b0f3506d5fdadb1e30407df49bc26e7547a635e21852c28af2574f9bf0ea95e"}
Jul 13 15:03:10.001367 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:10.001341    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p" event={"ID":"c6765fa4-2372-4092-bc64-2601b628519a","Type":"ContainerStarted","Data":"86f4550e96aa4dd6c77ef12da92b966900568b0621682211b229e8663bbcd41c"}
Jul 13 15:03:10.032605 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:10.032551    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p" podStartSLOduration=1.815164518 podStartE2EDuration="2.032534142s" totalImagesPullingTime="217.369624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:03:10.030661375 +0000 UTC m=+616.227814322" watchObservedRunningTime="2026-07-13 15:03:10.032534142 +0000 UTC m=+616.229687089"
Jul 13 15:03:19.297863 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:19.297768    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:19.297863 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:19.297843    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:19.310239 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:19.310214    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:03:20.057378 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:03:20.057345    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p"
Jul 13 15:05:20.430530 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:20.430497    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c7db64446-4k2n7"]
Jul 13 15:05:20.433791 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:20.433770    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c7db64446-4k2n7"
Jul 13 15:05:20.436178 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:20.436158    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 13 15:05:20.436941 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:20.436926    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xhtrg\""
Jul 13 15:05:20.443610 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:20.443587    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c7db64446-4k2n7"]
Jul 13 15:05:20.528853 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:20.528811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/59192a8a-42f2-45f5-b8a4-1bfe09292ef7-tls-cert\") pod \"authorino-7c7db64446-4k2n7\" (UID: \"59192a8a-42f2-45f5-b8a4-1bfe09292ef7\") " pod="kuadrant-system/authorino-7c7db64446-4k2n7"
Jul 13 15:05:20.529027 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:20.528887    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22swd\" (UniqueName: \"kubernetes.io/projected/59192a8a-42f2-45f5-b8a4-1bfe09292ef7-kube-api-access-22swd\") pod \"authorino-7c7db64446-4k2n7\" (UID: \"59192a8a-42f2-45f5-b8a4-1bfe09292ef7\") " pod="kuadrant-system/authorino-7c7db64446-4k2n7"
Jul 13 15:05:20.629596 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:20.629558    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/59192a8a-42f2-45f5-b8a4-1bfe09292ef7-tls-cert\") pod \"authorino-7c7db64446-4k2n7\" (UID: \"59192a8a-42f2-45f5-b8a4-1bfe09292ef7\") " pod="kuadrant-system/authorino-7c7db64446-4k2n7"
Jul 13 15:05:20.629800 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:20.629638    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22swd\" (UniqueName: \"kubernetes.io/projected/59192a8a-42f2-45f5-b8a4-1bfe09292ef7-kube-api-access-22swd\") pod \"authorino-7c7db64446-4k2n7\" (UID: \"59192a8a-42f2-45f5-b8a4-1bfe09292ef7\") " pod="kuadrant-system/authorino-7c7db64446-4k2n7"
Jul 13 15:05:20.632152 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:20.632127    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/59192a8a-42f2-45f5-b8a4-1bfe09292ef7-tls-cert\") pod \"authorino-7c7db64446-4k2n7\" (UID: \"59192a8a-42f2-45f5-b8a4-1bfe09292ef7\") " pod="kuadrant-system/authorino-7c7db64446-4k2n7"
Jul 13 15:05:20.639237 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:20.639212    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22swd\" (UniqueName: \"kubernetes.io/projected/59192a8a-42f2-45f5-b8a4-1bfe09292ef7-kube-api-access-22swd\") pod \"authorino-7c7db64446-4k2n7\" (UID: \"59192a8a-42f2-45f5-b8a4-1bfe09292ef7\") " pod="kuadrant-system/authorino-7c7db64446-4k2n7"
Jul 13 15:05:20.742942 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:20.742863    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c7db64446-4k2n7"
Jul 13 15:05:20.889302 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:20.889066    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c7db64446-4k2n7"]
Jul 13 15:05:20.891582 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:05:20.891554    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59192a8a_42f2_45f5_b8a4_1bfe09292ef7.slice/crio-6a338026e88fb2859e6947cb2409fbbd895ccc28605afde53f3dba56579b32d3 WatchSource:0}: Error finding container 6a338026e88fb2859e6947cb2409fbbd895ccc28605afde53f3dba56579b32d3: Status 404 returned error can't find the container with id 6a338026e88fb2859e6947cb2409fbbd895ccc28605afde53f3dba56579b32d3
Jul 13 15:05:21.575104 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:21.575070    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c7db64446-4k2n7" event={"ID":"59192a8a-42f2-45f5-b8a4-1bfe09292ef7","Type":"ContainerStarted","Data":"08a98c0c99de56c3a375cac46aaeb5ed3af2dc2d6db6bf31bf0bfec5a7e94863"}
Jul 13 15:05:21.575104 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:21.575115    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c7db64446-4k2n7" event={"ID":"59192a8a-42f2-45f5-b8a4-1bfe09292ef7","Type":"ContainerStarted","Data":"6a338026e88fb2859e6947cb2409fbbd895ccc28605afde53f3dba56579b32d3"}
Jul 13 15:05:21.591168 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:05:21.591029    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c7db64446-4k2n7" podStartSLOduration=0.987731956 podStartE2EDuration="1.591007379s" totalImagesPullingTime="603.275423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:05:21.590671517 +0000 UTC m=+747.787824465" watchObservedRunningTime="2026-07-13 15:05:21.591007379 +0000 UTC m=+747.788160336"
Jul 13 15:07:54.247610 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:07:54.247584    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:07:54.248079 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:07:54.247864    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:12:54.275424 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:12:54.275397    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:12:54.276959 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:12:54.276933    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:17:54.303554 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:17:54.303524    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:17:54.306143 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:17:54.304590    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:18:47.388229 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:47.388194    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7"]
Jul 13 15:18:47.391969 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:47.391950    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7"
Jul 13 15:18:47.394424 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:47.394402    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 15:18:47.394522 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:47.394435    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\""
Jul 13 15:18:47.399878 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:47.399281    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7"]
Jul 13 15:18:47.484951 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:47.484915    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7\" (UID: \"67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7"
Jul 13 15:18:47.585924 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:47.585889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7\" (UID: \"67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7"
Jul 13 15:18:47.586593 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:47.586572    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7\" (UID: \"67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7"
Jul 13 15:18:47.703718 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:47.703668    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7"
Jul 13 15:18:47.839482 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:47.839458    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7"]
Jul 13 15:18:47.842081 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:18:47.842046    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ff49c2_f7f8_4651_b7ce_c6a1b70c34aa.slice/crio-c68e7ba7e3b79c94bfd52492595801d4205e23217e7bc86d9030d8fb0996dcb2 WatchSource:0}: Error finding container c68e7ba7e3b79c94bfd52492595801d4205e23217e7bc86d9030d8fb0996dcb2: Status 404 returned error can't find the container with id c68e7ba7e3b79c94bfd52492595801d4205e23217e7bc86d9030d8fb0996dcb2
Jul 13 15:18:47.843928 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:47.843911    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 15:18:48.215576 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:48.215535    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7" event={"ID":"67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa","Type":"ContainerStarted","Data":"c68e7ba7e3b79c94bfd52492595801d4205e23217e7bc86d9030d8fb0996dcb2"}
Jul 13 15:18:50.227006 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:50.226971    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7" event={"ID":"67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa","Type":"ContainerStarted","Data":"ad2f69f649d35fbc3a65073666ed7d2a4d71ba3dde2cabfa34fadb749bcfffa7"}
Jul 13 15:18:50.246146 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:50.246088    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7" podStartSLOduration=1.3540181869999999 podStartE2EDuration="3.246072112s" totalImagesPullingTime="1.892053925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:18:50.244452665 +0000 UTC m=+1556.441605616" watchObservedRunningTime="2026-07-13 15:18:50.246072112 +0000 UTC m=+1556.443225060"
Jul 13 15:18:51.234007 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:51.233969    2564 generic.go:356] "Generic (PLEG): container finished" podID="67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa" containerID="ad2f69f649d35fbc3a65073666ed7d2a4d71ba3dde2cabfa34fadb749bcfffa7" exitCode=7
Jul 13 15:18:51.234492 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:51.234062    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7" event={"ID":"67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa","Type":"ContainerDied","Data":"ad2f69f649d35fbc3a65073666ed7d2a4d71ba3dde2cabfa34fadb749bcfffa7"}
Jul 13 15:18:51.234492 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:51.234240    2564 scope.go:117] "RemoveContainer" containerID="ad2f69f649d35fbc3a65073666ed7d2a4d71ba3dde2cabfa34fadb749bcfffa7"
Jul 13 15:18:52.240166 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:52.240079    2564 generic.go:356] "Generic (PLEG): container finished" podID="67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa" containerID="241a79a4bfee0f115bdaefeaad442aec572fa3d3cfeafefc38a3fadc9c002642" exitCode=0
Jul 13 15:18:52.240166 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:52.240128    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7" event={"ID":"67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa","Type":"ContainerDied","Data":"241a79a4bfee0f115bdaefeaad442aec572fa3d3cfeafefc38a3fadc9c002642"}
Jul 13 15:18:52.240595 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:52.240202    2564 scope.go:117] "RemoveContainer" containerID="ad2f69f649d35fbc3a65073666ed7d2a4d71ba3dde2cabfa34fadb749bcfffa7"
Jul 13 15:18:53.375961 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:53.375938    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7"
Jul 13 15:18:53.437945 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:53.437916    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa-maas-api-service-ca\") pod \"67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa\" (UID: \"67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa\") "
Jul 13 15:18:53.438302 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:53.438277    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa" (UID: "67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:18:53.539394 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:53.539307    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa-maas-api-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:18:54.252088 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:54.252054    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7"
Jul 13 15:18:54.252252 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:54.252092    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7" event={"ID":"67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa","Type":"ContainerDied","Data":"c68e7ba7e3b79c94bfd52492595801d4205e23217e7bc86d9030d8fb0996dcb2"}
Jul 13 15:18:54.252252 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:18:54.252140    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c68e7ba7e3b79c94bfd52492595801d4205e23217e7bc86d9030d8fb0996dcb2"
Jul 13 15:19:14.621377 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.621338    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"]
Jul 13 15:19:14.621889 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.621669    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa" containerName="revoke-keys"
Jul 13 15:19:14.621889 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.621680    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa" containerName="revoke-keys"
Jul 13 15:19:14.621889 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.621722    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa" containerName="revoke-keys"
Jul 13 15:19:14.621889 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.621729    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa" containerName="revoke-keys"
Jul 13 15:19:14.621889 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.621798    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa" containerName="revoke-keys"
Jul 13 15:19:14.621889 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.621808    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa" containerName="revoke-keys"
Jul 13 15:19:14.629475 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.629449    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:19:14.631998 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.631969    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 15:19:14.632144 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.631966    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-15019914-serving-cert\""
Jul 13 15:19:14.632144 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.632067    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 15:19:14.632954 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.632934    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvc6k\""
Jul 13 15:19:14.637241 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.637213    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"]
Jul 13 15:19:14.722080 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.722045    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b2dfd-441e-47ce-b070-daca27bb540c-openshift-service-ca\") pod \"maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm\" (UID: \"ff6b2dfd-441e-47ce-b070-daca27bb540c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:19:14.722255 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.722107    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff6b2dfd-441e-47ce-b070-daca27bb540c-maas-api-tls\") pod \"maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm\" (UID: \"ff6b2dfd-441e-47ce-b070-daca27bb540c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:19:14.722255 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.722188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np6cd\" (UniqueName: \"kubernetes.io/projected/ff6b2dfd-441e-47ce-b070-daca27bb540c-kube-api-access-np6cd\") pod \"maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm\" (UID: \"ff6b2dfd-441e-47ce-b070-daca27bb540c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:19:14.822895 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.822857    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff6b2dfd-441e-47ce-b070-daca27bb540c-maas-api-tls\") pod \"maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm\" (UID: \"ff6b2dfd-441e-47ce-b070-daca27bb540c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:19:14.823066 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.822931    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np6cd\" (UniqueName: \"kubernetes.io/projected/ff6b2dfd-441e-47ce-b070-daca27bb540c-kube-api-access-np6cd\") pod \"maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm\" (UID: \"ff6b2dfd-441e-47ce-b070-daca27bb540c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:19:14.823066 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.822957    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b2dfd-441e-47ce-b070-daca27bb540c-openshift-service-ca\") pod \"maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm\" (UID: \"ff6b2dfd-441e-47ce-b070-daca27bb540c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:19:14.823554 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.823536    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b2dfd-441e-47ce-b070-daca27bb540c-openshift-service-ca\") pod \"maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm\" (UID: \"ff6b2dfd-441e-47ce-b070-daca27bb540c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:19:14.825426 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.825409    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff6b2dfd-441e-47ce-b070-daca27bb540c-maas-api-tls\") pod \"maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm\" (UID: \"ff6b2dfd-441e-47ce-b070-daca27bb540c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:19:14.831917 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.831887    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np6cd\" (UniqueName: \"kubernetes.io/projected/ff6b2dfd-441e-47ce-b070-daca27bb540c-kube-api-access-np6cd\") pod \"maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm\" (UID: \"ff6b2dfd-441e-47ce-b070-daca27bb540c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:19:14.944216 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:14.944183    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:19:15.084236 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:15.084213    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"]
Jul 13 15:19:15.086440 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:19:15.086412    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6b2dfd_441e_47ce_b070_daca27bb540c.slice/crio-788d5e20ccb7cf66b01117e37b5c29b947a0a4c82af3ffec8b937adbe0bc748f WatchSource:0}: Error finding container 788d5e20ccb7cf66b01117e37b5c29b947a0a4c82af3ffec8b937adbe0bc748f: Status 404 returned error can't find the container with id 788d5e20ccb7cf66b01117e37b5c29b947a0a4c82af3ffec8b937adbe0bc748f
Jul 13 15:19:15.345790 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:15.345679    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm" event={"ID":"ff6b2dfd-441e-47ce-b070-daca27bb540c","Type":"ContainerStarted","Data":"788d5e20ccb7cf66b01117e37b5c29b947a0a4c82af3ffec8b937adbe0bc748f"}
Jul 13 15:19:17.357439 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:17.357391    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm" event={"ID":"ff6b2dfd-441e-47ce-b070-daca27bb540c","Type":"ContainerStarted","Data":"d42300aae0a9910f58bc06cc4d1a5113cc3433e4c3b8d8d5fc2e91eee32c3091"}
Jul 13 15:19:17.357994 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:17.357456    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:19:17.377437 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:17.377386    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm" podStartSLOduration=1.25246055 podStartE2EDuration="3.377368105s" totalImagesPullingTime="2.124907555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:19:17.376176084 +0000 UTC m=+1583.573329035" watchObservedRunningTime="2026-07-13 15:19:17.377368105 +0000 UTC m=+1583.574521053"
Jul 13 15:19:23.367559 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:19:23.367483    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:20:19.171610 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:19.171574    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-mrbs6"]
Jul 13 15:20:19.174309 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:19.174286    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-mrbs6"
Jul 13 15:20:19.177424 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:19.177405    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\""
Jul 13 15:20:19.186035 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:19.186014    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-mrbs6"]
Jul 13 15:20:19.194775 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:19.194737    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29e0619-2a2c-44af-b276-8880be8db05e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-15019914-mrbs6\" (UID: \"d29e0619-2a2c-44af-b276-8880be8db05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-mrbs6"
Jul 13 15:20:19.295975 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:19.295937    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29e0619-2a2c-44af-b276-8880be8db05e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-15019914-mrbs6\" (UID: \"d29e0619-2a2c-44af-b276-8880be8db05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-mrbs6"
Jul 13 15:20:19.296544 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:19.296524    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29e0619-2a2c-44af-b276-8880be8db05e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-15019914-mrbs6\" (UID: \"d29e0619-2a2c-44af-b276-8880be8db05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-mrbs6"
Jul 13 15:20:19.484641 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:19.484564    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-mrbs6"
Jul 13 15:20:19.825415 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:19.825392    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-mrbs6"]
Jul 13 15:20:19.827895 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:20:19.827866    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29e0619_2a2c_44af_b276_8880be8db05e.slice/crio-0f966c68c36e5c69d9d487eda1362adbc3658fcd90641515c95aea783fc509db WatchSource:0}: Error finding container 0f966c68c36e5c69d9d487eda1362adbc3658fcd90641515c95aea783fc509db: Status 404 returned error can't find the container with id 0f966c68c36e5c69d9d487eda1362adbc3658fcd90641515c95aea783fc509db
Jul 13 15:20:20.650188 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:20.650150    2564 generic.go:356] "Generic (PLEG): container finished" podID="d29e0619-2a2c-44af-b276-8880be8db05e" containerID="e3e88d2b8da86944c944bb786c88b1ef10b75d9905b3177073fc17fa3f22284b" exitCode=0
Jul 13 15:20:20.650539 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:20.650233    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-mrbs6" event={"ID":"d29e0619-2a2c-44af-b276-8880be8db05e","Type":"ContainerDied","Data":"e3e88d2b8da86944c944bb786c88b1ef10b75d9905b3177073fc17fa3f22284b"}
Jul 13 15:20:20.650539 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:20.650279    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-mrbs6" event={"ID":"d29e0619-2a2c-44af-b276-8880be8db05e","Type":"ContainerStarted","Data":"0f966c68c36e5c69d9d487eda1362adbc3658fcd90641515c95aea783fc509db"}
Jul 13 15:20:21.783385 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:21.783361    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-mrbs6"
Jul 13 15:20:21.812045 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:21.812013    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29e0619-2a2c-44af-b276-8880be8db05e-maas-api-service-ca\") pod \"d29e0619-2a2c-44af-b276-8880be8db05e\" (UID: \"d29e0619-2a2c-44af-b276-8880be8db05e\") "
Jul 13 15:20:21.812340 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:21.812319    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29e0619-2a2c-44af-b276-8880be8db05e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d29e0619-2a2c-44af-b276-8880be8db05e" (UID: "d29e0619-2a2c-44af-b276-8880be8db05e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:20:21.913262 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:21.913183    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29e0619-2a2c-44af-b276-8880be8db05e-maas-api-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:20:22.660511 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:22.660487    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-mrbs6"
Jul 13 15:20:22.660669 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:22.660518    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-15019914-mrbs6" event={"ID":"d29e0619-2a2c-44af-b276-8880be8db05e","Type":"ContainerDied","Data":"0f966c68c36e5c69d9d487eda1362adbc3658fcd90641515c95aea783fc509db"}
Jul 13 15:20:22.660669 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:22.660561    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f966c68c36e5c69d9d487eda1362adbc3658fcd90641515c95aea783fc509db"
Jul 13 15:20:25.391859 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.391824    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"]
Jul 13 15:20:25.392546 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.392510    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm" podUID="ff6b2dfd-441e-47ce-b070-daca27bb540c" containerName="maas-api" containerID="cri-o://d42300aae0a9910f58bc06cc4d1a5113cc3433e4c3b8d8d5fc2e91eee32c3091" gracePeriod=30
Jul 13 15:20:25.659474 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.659452    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:20:25.674481 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.674445    2564 generic.go:356] "Generic (PLEG): container finished" podID="ff6b2dfd-441e-47ce-b070-daca27bb540c" containerID="d42300aae0a9910f58bc06cc4d1a5113cc3433e4c3b8d8d5fc2e91eee32c3091" exitCode=0
Jul 13 15:20:25.674625 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.674483    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"
Jul 13 15:20:25.674625 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.674537    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm" event={"ID":"ff6b2dfd-441e-47ce-b070-daca27bb540c","Type":"ContainerDied","Data":"d42300aae0a9910f58bc06cc4d1a5113cc3433e4c3b8d8d5fc2e91eee32c3091"}
Jul 13 15:20:25.674625 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.674602    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm" event={"ID":"ff6b2dfd-441e-47ce-b070-daca27bb540c","Type":"ContainerDied","Data":"788d5e20ccb7cf66b01117e37b5c29b947a0a4c82af3ffec8b937adbe0bc748f"}
Jul 13 15:20:25.674831 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.674632    2564 scope.go:117] "RemoveContainer" containerID="d42300aae0a9910f58bc06cc4d1a5113cc3433e4c3b8d8d5fc2e91eee32c3091"
Jul 13 15:20:25.684656 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.684637    2564 scope.go:117] "RemoveContainer" containerID="d42300aae0a9910f58bc06cc4d1a5113cc3433e4c3b8d8d5fc2e91eee32c3091"
Jul 13 15:20:25.684991 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:20:25.684971    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42300aae0a9910f58bc06cc4d1a5113cc3433e4c3b8d8d5fc2e91eee32c3091\": container with ID starting with d42300aae0a9910f58bc06cc4d1a5113cc3433e4c3b8d8d5fc2e91eee32c3091 not found: ID does not exist" containerID="d42300aae0a9910f58bc06cc4d1a5113cc3433e4c3b8d8d5fc2e91eee32c3091"
Jul 13 15:20:25.685111 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.685000    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42300aae0a9910f58bc06cc4d1a5113cc3433e4c3b8d8d5fc2e91eee32c3091"} err="failed to get container status \"d42300aae0a9910f58bc06cc4d1a5113cc3433e4c3b8d8d5fc2e91eee32c3091\": rpc error: code = NotFound desc = could not find container \"d42300aae0a9910f58bc06cc4d1a5113cc3433e4c3b8d8d5fc2e91eee32c3091\": container with ID starting with d42300aae0a9910f58bc06cc4d1a5113cc3433e4c3b8d8d5fc2e91eee32c3091 not found: ID does not exist"
Jul 13 15:20:25.745071 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.745031    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff6b2dfd-441e-47ce-b070-daca27bb540c-maas-api-tls\") pod \"ff6b2dfd-441e-47ce-b070-daca27bb540c\" (UID: \"ff6b2dfd-441e-47ce-b070-daca27bb540c\") "
Jul 13 15:20:25.745272 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.745084    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np6cd\" (UniqueName: \"kubernetes.io/projected/ff6b2dfd-441e-47ce-b070-daca27bb540c-kube-api-access-np6cd\") pod \"ff6b2dfd-441e-47ce-b070-daca27bb540c\" (UID: \"ff6b2dfd-441e-47ce-b070-daca27bb540c\") "
Jul 13 15:20:25.745272 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.745115    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b2dfd-441e-47ce-b070-daca27bb540c-openshift-service-ca\") pod \"ff6b2dfd-441e-47ce-b070-daca27bb540c\" (UID: \"ff6b2dfd-441e-47ce-b070-daca27bb540c\") "
Jul 13 15:20:25.745546 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.745516    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6b2dfd-441e-47ce-b070-daca27bb540c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff6b2dfd-441e-47ce-b070-daca27bb540c" (UID: "ff6b2dfd-441e-47ce-b070-daca27bb540c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:20:25.747529 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.747500    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6b2dfd-441e-47ce-b070-daca27bb540c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff6b2dfd-441e-47ce-b070-daca27bb540c" (UID: "ff6b2dfd-441e-47ce-b070-daca27bb540c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:20:25.747529 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.747513    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6b2dfd-441e-47ce-b070-daca27bb540c-kube-api-access-np6cd" (OuterVolumeSpecName: "kube-api-access-np6cd") pod "ff6b2dfd-441e-47ce-b070-daca27bb540c" (UID: "ff6b2dfd-441e-47ce-b070-daca27bb540c"). InnerVolumeSpecName "kube-api-access-np6cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:20:25.846395 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.846354    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff6b2dfd-441e-47ce-b070-daca27bb540c-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:20:25.846395 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.846388    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np6cd\" (UniqueName: \"kubernetes.io/projected/ff6b2dfd-441e-47ce-b070-daca27bb540c-kube-api-access-np6cd\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:20:25.846395 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.846397    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b2dfd-441e-47ce-b070-daca27bb540c-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:20:25.997818 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:25.997789    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"]
Jul 13 15:20:26.002727 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:26.002689    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-15019914-f7f8d97c-p2rvm"]
Jul 13 15:20:26.316742 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:26.316656    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff6b2dfd-441e-47ce-b070-daca27bb540c" path="/var/lib/kubelet/pods/ff6b2dfd-441e-47ce-b070-daca27bb540c/volumes"
Jul 13 15:20:33.217552 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.217518    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"]
Jul 13 15:20:33.217933 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.217866    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6b2dfd-441e-47ce-b070-daca27bb540c" containerName="maas-api"
Jul 13 15:20:33.217933 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.217878    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6b2dfd-441e-47ce-b070-daca27bb540c" containerName="maas-api"
Jul 13 15:20:33.217933 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.217900    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29e0619-2a2c-44af-b276-8880be8db05e" containerName="revoke-keys"
Jul 13 15:20:33.217933 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.217905    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29e0619-2a2c-44af-b276-8880be8db05e" containerName="revoke-keys"
Jul 13 15:20:33.218057 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.217961    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6b2dfd-441e-47ce-b070-daca27bb540c" containerName="maas-api"
Jul 13 15:20:33.218057 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.217971    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29e0619-2a2c-44af-b276-8880be8db05e" containerName="revoke-keys"
Jul 13 15:20:33.221112 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.221095    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:20:33.223850 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.223827    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvc6k\""
Jul 13 15:20:33.223956 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.223849    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 15:20:33.224712 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.224678    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-76c770f3-serving-cert\""
Jul 13 15:20:33.224819 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.224752    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 15:20:33.234068 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.234041    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"]
Jul 13 15:20:33.313447 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.313420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce3b554-436a-427b-a8a9-ce2972b817d8-maas-api-tls\") pod \"maas-api-e2e-ait-76c770f3-5446765887-jgwfb\" (UID: \"9ce3b554-436a-427b-a8a9-ce2972b817d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:20:33.313447 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.313451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvk59\" (UniqueName: \"kubernetes.io/projected/9ce3b554-436a-427b-a8a9-ce2972b817d8-kube-api-access-rvk59\") pod \"maas-api-e2e-ait-76c770f3-5446765887-jgwfb\" (UID: \"9ce3b554-436a-427b-a8a9-ce2972b817d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:20:33.313635 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.313490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce3b554-436a-427b-a8a9-ce2972b817d8-openshift-service-ca\") pod \"maas-api-e2e-ait-76c770f3-5446765887-jgwfb\" (UID: \"9ce3b554-436a-427b-a8a9-ce2972b817d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:20:33.414263 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.414229    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce3b554-436a-427b-a8a9-ce2972b817d8-maas-api-tls\") pod \"maas-api-e2e-ait-76c770f3-5446765887-jgwfb\" (UID: \"9ce3b554-436a-427b-a8a9-ce2972b817d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:20:33.414430 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.414275    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvk59\" (UniqueName: \"kubernetes.io/projected/9ce3b554-436a-427b-a8a9-ce2972b817d8-kube-api-access-rvk59\") pod \"maas-api-e2e-ait-76c770f3-5446765887-jgwfb\" (UID: \"9ce3b554-436a-427b-a8a9-ce2972b817d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:20:33.414430 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:20:33.414376    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-serving-cert: secret "maas-api-e2e-ait-76c770f3-serving-cert" not found
Jul 13 15:20:33.414517 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:20:33.414435    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce3b554-436a-427b-a8a9-ce2972b817d8-maas-api-tls podName:9ce3b554-436a-427b-a8a9-ce2972b817d8 nodeName:}" failed. No retries permitted until 2026-07-13 15:20:33.914419076 +0000 UTC m=+1660.111572003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ce3b554-436a-427b-a8a9-ce2972b817d8-maas-api-tls") pod "maas-api-e2e-ait-76c770f3-5446765887-jgwfb" (UID: "9ce3b554-436a-427b-a8a9-ce2972b817d8") : secret "maas-api-e2e-ait-76c770f3-serving-cert" not found
Jul 13 15:20:33.414517 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.414466    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce3b554-436a-427b-a8a9-ce2972b817d8-openshift-service-ca\") pod \"maas-api-e2e-ait-76c770f3-5446765887-jgwfb\" (UID: \"9ce3b554-436a-427b-a8a9-ce2972b817d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:20:33.415088 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.415072    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce3b554-436a-427b-a8a9-ce2972b817d8-openshift-service-ca\") pod \"maas-api-e2e-ait-76c770f3-5446765887-jgwfb\" (UID: \"9ce3b554-436a-427b-a8a9-ce2972b817d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:20:33.424007 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.423987    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvk59\" (UniqueName: \"kubernetes.io/projected/9ce3b554-436a-427b-a8a9-ce2972b817d8-kube-api-access-rvk59\") pod \"maas-api-e2e-ait-76c770f3-5446765887-jgwfb\" (UID: \"9ce3b554-436a-427b-a8a9-ce2972b817d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:20:33.918671 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.918631    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce3b554-436a-427b-a8a9-ce2972b817d8-maas-api-tls\") pod \"maas-api-e2e-ait-76c770f3-5446765887-jgwfb\" (UID: \"9ce3b554-436a-427b-a8a9-ce2972b817d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:20:33.921206 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:33.921173    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce3b554-436a-427b-a8a9-ce2972b817d8-maas-api-tls\") pod \"maas-api-e2e-ait-76c770f3-5446765887-jgwfb\" (UID: \"9ce3b554-436a-427b-a8a9-ce2972b817d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:20:34.131725 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:34.131658    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:20:34.270609 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:34.270577    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"]
Jul 13 15:20:34.273184 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:20:34.273147    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce3b554_436a_427b_a8a9_ce2972b817d8.slice/crio-b5d24b3e1f1b016cbf8f71095265813eff51be13ca3e13b49caf56ee288eea52 WatchSource:0}: Error finding container b5d24b3e1f1b016cbf8f71095265813eff51be13ca3e13b49caf56ee288eea52: Status 404 returned error can't find the container with id b5d24b3e1f1b016cbf8f71095265813eff51be13ca3e13b49caf56ee288eea52
Jul 13 15:20:34.716604 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:34.716578    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb" event={"ID":"9ce3b554-436a-427b-a8a9-ce2972b817d8","Type":"ContainerStarted","Data":"b5d24b3e1f1b016cbf8f71095265813eff51be13ca3e13b49caf56ee288eea52"}
Jul 13 15:20:35.722546 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:35.722505    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb" event={"ID":"9ce3b554-436a-427b-a8a9-ce2972b817d8","Type":"ContainerStarted","Data":"583c753c853acef3a9e12102d7209232266387dfc1121fa4fdca17dcbaf6b7fa"}
Jul 13 15:20:35.722981 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:35.722607    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:20:35.741728 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:35.741650    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb" podStartSLOduration=2.309182636 podStartE2EDuration="2.741631374s" totalImagesPullingTime="432.448738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:20:35.740867505 +0000 UTC m=+1661.938020455" watchObservedRunningTime="2026-07-13 15:20:35.741631374 +0000 UTC m=+1661.938784325"
Jul 13 15:20:38.289092 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:38.289063    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5"]
Jul 13 15:20:38.292632 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:38.292611    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5"
Jul 13 15:20:38.295146 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:38.295125    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\""
Jul 13 15:20:38.301819 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:38.301792    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5"]
Jul 13 15:20:38.359633 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:38.359596    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/072e8b23-d873-453e-b913-d758ff232830-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5\" (UID: \"072e8b23-d873-453e-b913-d758ff232830\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5"
Jul 13 15:20:38.460962 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:38.460927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/072e8b23-d873-453e-b913-d758ff232830-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5\" (UID: \"072e8b23-d873-453e-b913-d758ff232830\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5"
Jul 13 15:20:38.461596 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:38.461571    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/072e8b23-d873-453e-b913-d758ff232830-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5\" (UID: \"072e8b23-d873-453e-b913-d758ff232830\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5"
Jul 13 15:20:38.604068 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:38.603984    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5"
Jul 13 15:20:38.734719 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:38.734656    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5"]
Jul 13 15:20:38.738625 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:20:38.738590    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072e8b23_d873_453e_b913_d758ff232830.slice/crio-2a7104094472d22c5a8d45e218f3c70ca0facc0a751518c8f68ffeceb165d4dd WatchSource:0}: Error finding container 2a7104094472d22c5a8d45e218f3c70ca0facc0a751518c8f68ffeceb165d4dd: Status 404 returned error can't find the container with id 2a7104094472d22c5a8d45e218f3c70ca0facc0a751518c8f68ffeceb165d4dd
Jul 13 15:20:39.743019 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:39.742989    2564 generic.go:356] "Generic (PLEG): container finished" podID="072e8b23-d873-453e-b913-d758ff232830" containerID="72e78299abd2ba9884b35e68b794e0bd020a45e7856c46aa24aedf3928a880e3" exitCode=7
Jul 13 15:20:39.743394 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:39.743082    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5" event={"ID":"072e8b23-d873-453e-b913-d758ff232830","Type":"ContainerDied","Data":"72e78299abd2ba9884b35e68b794e0bd020a45e7856c46aa24aedf3928a880e3"}
Jul 13 15:20:39.743394 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:39.743135    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5" event={"ID":"072e8b23-d873-453e-b913-d758ff232830","Type":"ContainerStarted","Data":"2a7104094472d22c5a8d45e218f3c70ca0facc0a751518c8f68ffeceb165d4dd"}
Jul 13 15:20:39.743394 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:39.743236    2564 scope.go:117] "RemoveContainer" containerID="72e78299abd2ba9884b35e68b794e0bd020a45e7856c46aa24aedf3928a880e3"
Jul 13 15:20:40.750030 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:40.749932    2564 generic.go:356] "Generic (PLEG): container finished" podID="072e8b23-d873-453e-b913-d758ff232830" containerID="9cac14a5ec6c7e03edfc32c1e14cdd26caf5154bc38947b620b036363f9047d7" exitCode=7
Jul 13 15:20:40.750030 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:40.749997    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5" event={"ID":"072e8b23-d873-453e-b913-d758ff232830","Type":"ContainerDied","Data":"9cac14a5ec6c7e03edfc32c1e14cdd26caf5154bc38947b620b036363f9047d7"}
Jul 13 15:20:40.750462 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:40.750040    2564 scope.go:117] "RemoveContainer" containerID="72e78299abd2ba9884b35e68b794e0bd020a45e7856c46aa24aedf3928a880e3"
Jul 13 15:20:40.750462 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:40.750190    2564 scope.go:117] "RemoveContainer" containerID="9cac14a5ec6c7e03edfc32c1e14cdd26caf5154bc38947b620b036363f9047d7"
Jul 13 15:20:40.750462 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:20:40.750433    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5_odh-ai-gateway-infra(072e8b23-d873-453e-b913-d758ff232830)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5" podUID="072e8b23-d873-453e-b913-d758ff232830"
Jul 13 15:20:41.732492 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:41.732467    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:20:41.758168 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:41.758136    2564 scope.go:117] "RemoveContainer" containerID="9cac14a5ec6c7e03edfc32c1e14cdd26caf5154bc38947b620b036363f9047d7"
Jul 13 15:20:41.758779 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:20:41.758321    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5_odh-ai-gateway-infra(072e8b23-d873-453e-b913-d758ff232830)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5" podUID="072e8b23-d873-453e-b913-d758ff232830"
Jul 13 15:20:53.311145 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:53.311118    2564 scope.go:117] "RemoveContainer" containerID="9cac14a5ec6c7e03edfc32c1e14cdd26caf5154bc38947b620b036363f9047d7"
Jul 13 15:20:53.816907 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:53.816872    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5" event={"ID":"072e8b23-d873-453e-b913-d758ff232830","Type":"ContainerStarted","Data":"35c3cbd7e7d804ae148e8d26ad18619cd5cec1cebedeb39091817c8629f721fa"}
Jul 13 15:20:53.835246 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:53.835187    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5" podStartSLOduration=15.835168183 podStartE2EDuration="15.835168183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:20:53.833619954 +0000 UTC m=+1680.030772907" watchObservedRunningTime="2026-07-13 15:20:53.835168183 +0000 UTC m=+1680.032321133"
Jul 13 15:20:54.822748 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:54.822683    2564 generic.go:356] "Generic (PLEG): container finished" podID="072e8b23-d873-453e-b913-d758ff232830" containerID="35c3cbd7e7d804ae148e8d26ad18619cd5cec1cebedeb39091817c8629f721fa" exitCode=0
Jul 13 15:20:54.823152 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:54.822753    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5" event={"ID":"072e8b23-d873-453e-b913-d758ff232830","Type":"ContainerDied","Data":"35c3cbd7e7d804ae148e8d26ad18619cd5cec1cebedeb39091817c8629f721fa"}
Jul 13 15:20:54.823152 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:54.822817    2564 scope.go:117] "RemoveContainer" containerID="9cac14a5ec6c7e03edfc32c1e14cdd26caf5154bc38947b620b036363f9047d7"
Jul 13 15:20:54.860650 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:54.860619    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5"]
Jul 13 15:20:55.966855 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:55.966823    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5"
Jul 13 15:20:56.131392 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:56.131297    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/072e8b23-d873-453e-b913-d758ff232830-maas-api-service-ca\") pod \"072e8b23-d873-453e-b913-d758ff232830\" (UID: \"072e8b23-d873-453e-b913-d758ff232830\") "
Jul 13 15:20:56.131738 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:56.131677    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072e8b23-d873-453e-b913-d758ff232830-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "072e8b23-d873-453e-b913-d758ff232830" (UID: "072e8b23-d873-453e-b913-d758ff232830"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:20:56.232833 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:56.232798    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/072e8b23-d873-453e-b913-d758ff232830-maas-api-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:20:56.835335 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:56.835298    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5" event={"ID":"072e8b23-d873-453e-b913-d758ff232830","Type":"ContainerDied","Data":"2a7104094472d22c5a8d45e218f3c70ca0facc0a751518c8f68ffeceb165d4dd"}
Jul 13 15:20:56.835503 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:56.835353    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a7104094472d22c5a8d45e218f3c70ca0facc0a751518c8f68ffeceb165d4dd"
Jul 13 15:20:56.835503 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:56.835313    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5"
Jul 13 15:20:56.868919 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:56.868888    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5"]
Jul 13 15:20:56.874992 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:56.874968    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-vtnc5"]
Jul 13 15:20:58.316804 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:20:58.316773    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="072e8b23-d873-453e-b913-d758ff232830" path="/var/lib/kubelet/pods/072e8b23-d873-453e-b913-d758ff232830/volumes"
Jul 13 15:21:28.355873 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.355782    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r"]
Jul 13 15:21:28.356436 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.356303    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072e8b23-d873-453e-b913-d758ff232830" containerName="revoke-keys"
Jul 13 15:21:28.356436 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.356323    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="072e8b23-d873-453e-b913-d758ff232830" containerName="revoke-keys"
Jul 13 15:21:28.356436 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.356346    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072e8b23-d873-453e-b913-d758ff232830" containerName="revoke-keys"
Jul 13 15:21:28.356436 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.356355    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="072e8b23-d873-453e-b913-d758ff232830" containerName="revoke-keys"
Jul 13 15:21:28.356436 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.356376    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072e8b23-d873-453e-b913-d758ff232830" containerName="revoke-keys"
Jul 13 15:21:28.356436 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.356385    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="072e8b23-d873-453e-b913-d758ff232830" containerName="revoke-keys"
Jul 13 15:21:28.356756 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.356479    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="072e8b23-d873-453e-b913-d758ff232830" containerName="revoke-keys"
Jul 13 15:21:28.356756 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.356491    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="072e8b23-d873-453e-b913-d758ff232830" containerName="revoke-keys"
Jul 13 15:21:28.359394 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.359372    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r"
Jul 13 15:21:28.363296 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.363269    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\""
Jul 13 15:21:28.371758 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.371731    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r"]
Jul 13 15:21:28.402819 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.402780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d816e2-bf78-49ad-a03d-7e0d8eda90f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r\" (UID: \"42d816e2-bf78-49ad-a03d-7e0d8eda90f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r"
Jul 13 15:21:28.503995 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.503947    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d816e2-bf78-49ad-a03d-7e0d8eda90f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r\" (UID: \"42d816e2-bf78-49ad-a03d-7e0d8eda90f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r"
Jul 13 15:21:28.504640 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.504618    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d816e2-bf78-49ad-a03d-7e0d8eda90f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r\" (UID: \"42d816e2-bf78-49ad-a03d-7e0d8eda90f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r"
Jul 13 15:21:28.670304 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.670269    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r"
Jul 13 15:21:28.820111 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.819977    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r"]
Jul 13 15:21:28.823111 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:21:28.823076    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d816e2_bf78_49ad_a03d_7e0d8eda90f8.slice/crio-595be24c1408230086bb83881dccc18827f902bbb12254c45930bba890d0f569 WatchSource:0}: Error finding container 595be24c1408230086bb83881dccc18827f902bbb12254c45930bba890d0f569: Status 404 returned error can't find the container with id 595be24c1408230086bb83881dccc18827f902bbb12254c45930bba890d0f569
Jul 13 15:21:28.987526 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:28.987440    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r" event={"ID":"42d816e2-bf78-49ad-a03d-7e0d8eda90f8","Type":"ContainerStarted","Data":"595be24c1408230086bb83881dccc18827f902bbb12254c45930bba890d0f569"}
Jul 13 15:21:29.993773 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:29.993730    2564 generic.go:356] "Generic (PLEG): container finished" podID="42d816e2-bf78-49ad-a03d-7e0d8eda90f8" containerID="935edf66a3cdf1d0116fa2c8e2f4e2e3939ce3884916657ff059442fb1dbea83" exitCode=0
Jul 13 15:21:29.994172 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:29.993868    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r" event={"ID":"42d816e2-bf78-49ad-a03d-7e0d8eda90f8","Type":"ContainerDied","Data":"935edf66a3cdf1d0116fa2c8e2f4e2e3939ce3884916657ff059442fb1dbea83"}
Jul 13 15:21:31.138711 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:31.138680    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r"
Jul 13 15:21:31.229206 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:31.229168    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d816e2-bf78-49ad-a03d-7e0d8eda90f8-maas-api-service-ca\") pod \"42d816e2-bf78-49ad-a03d-7e0d8eda90f8\" (UID: \"42d816e2-bf78-49ad-a03d-7e0d8eda90f8\") "
Jul 13 15:21:31.229600 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:31.229579    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d816e2-bf78-49ad-a03d-7e0d8eda90f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42d816e2-bf78-49ad-a03d-7e0d8eda90f8" (UID: "42d816e2-bf78-49ad-a03d-7e0d8eda90f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:21:31.330512 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:31.330423    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d816e2-bf78-49ad-a03d-7e0d8eda90f8-maas-api-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:21:32.004877 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:32.004848    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r"
Jul 13 15:21:32.005039 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:32.004854    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r" event={"ID":"42d816e2-bf78-49ad-a03d-7e0d8eda90f8","Type":"ContainerDied","Data":"595be24c1408230086bb83881dccc18827f902bbb12254c45930bba890d0f569"}
Jul 13 15:21:32.005039 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:32.004957    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="595be24c1408230086bb83881dccc18827f902bbb12254c45930bba890d0f569"
Jul 13 15:21:38.840096 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:38.840066    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"]
Jul 13 15:21:38.840494 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:38.840294    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb" podUID="9ce3b554-436a-427b-a8a9-ce2972b817d8" containerName="maas-api" containerID="cri-o://583c753c853acef3a9e12102d7209232266387dfc1121fa4fdca17dcbaf6b7fa" gracePeriod=30
Jul 13 15:21:39.039937 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:39.039902    2564 generic.go:356] "Generic (PLEG): container finished" podID="9ce3b554-436a-427b-a8a9-ce2972b817d8" containerID="583c753c853acef3a9e12102d7209232266387dfc1121fa4fdca17dcbaf6b7fa" exitCode=0
Jul 13 15:21:39.040106 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:39.039980    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb" event={"ID":"9ce3b554-436a-427b-a8a9-ce2972b817d8","Type":"ContainerDied","Data":"583c753c853acef3a9e12102d7209232266387dfc1121fa4fdca17dcbaf6b7fa"}
Jul 13 15:21:39.100626 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:39.100552    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:21:39.201664 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:39.201635    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce3b554-436a-427b-a8a9-ce2972b817d8-maas-api-tls\") pod \"9ce3b554-436a-427b-a8a9-ce2972b817d8\" (UID: \"9ce3b554-436a-427b-a8a9-ce2972b817d8\") "
Jul 13 15:21:39.201889 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:39.201761    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvk59\" (UniqueName: \"kubernetes.io/projected/9ce3b554-436a-427b-a8a9-ce2972b817d8-kube-api-access-rvk59\") pod \"9ce3b554-436a-427b-a8a9-ce2972b817d8\" (UID: \"9ce3b554-436a-427b-a8a9-ce2972b817d8\") "
Jul 13 15:21:39.201889 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:39.201797    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce3b554-436a-427b-a8a9-ce2972b817d8-openshift-service-ca\") pod \"9ce3b554-436a-427b-a8a9-ce2972b817d8\" (UID: \"9ce3b554-436a-427b-a8a9-ce2972b817d8\") "
Jul 13 15:21:39.202194 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:39.202169    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce3b554-436a-427b-a8a9-ce2972b817d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ce3b554-436a-427b-a8a9-ce2972b817d8" (UID: "9ce3b554-436a-427b-a8a9-ce2972b817d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:21:39.203973 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:39.203947    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce3b554-436a-427b-a8a9-ce2972b817d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ce3b554-436a-427b-a8a9-ce2972b817d8" (UID: "9ce3b554-436a-427b-a8a9-ce2972b817d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:21:39.204049 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:39.204007    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce3b554-436a-427b-a8a9-ce2972b817d8-kube-api-access-rvk59" (OuterVolumeSpecName: "kube-api-access-rvk59") pod "9ce3b554-436a-427b-a8a9-ce2972b817d8" (UID: "9ce3b554-436a-427b-a8a9-ce2972b817d8"). InnerVolumeSpecName "kube-api-access-rvk59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:21:39.303073 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:39.303033    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce3b554-436a-427b-a8a9-ce2972b817d8-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:21:39.303073 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:39.303067    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce3b554-436a-427b-a8a9-ce2972b817d8-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:21:39.303073 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:39.303077    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvk59\" (UniqueName: \"kubernetes.io/projected/9ce3b554-436a-427b-a8a9-ce2972b817d8-kube-api-access-rvk59\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:21:40.046017 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:40.045981    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"
Jul 13 15:21:40.046017 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:40.046012    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb" event={"ID":"9ce3b554-436a-427b-a8a9-ce2972b817d8","Type":"ContainerDied","Data":"b5d24b3e1f1b016cbf8f71095265813eff51be13ca3e13b49caf56ee288eea52"}
Jul 13 15:21:40.046522 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:40.046073    2564 scope.go:117] "RemoveContainer" containerID="583c753c853acef3a9e12102d7209232266387dfc1121fa4fdca17dcbaf6b7fa"
Jul 13 15:21:40.076295 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:40.076261    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"]
Jul 13 15:21:40.082687 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:40.082661    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76c770f3-5446765887-jgwfb"]
Jul 13 15:21:40.317180 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:40.317100    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ce3b554-436a-427b-a8a9-ce2972b817d8" path="/var/lib/kubelet/pods/9ce3b554-436a-427b-a8a9-ce2972b817d8/volumes"
Jul 13 15:21:56.498439 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.498407    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"]
Jul 13 15:21:56.498799 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.498789    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ce3b554-436a-427b-a8a9-ce2972b817d8" containerName="maas-api"
Jul 13 15:21:56.498852 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.498801    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ce3b554-436a-427b-a8a9-ce2972b817d8" containerName="maas-api"
Jul 13 15:21:56.498852 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.498821    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42d816e2-bf78-49ad-a03d-7e0d8eda90f8" containerName="revoke-keys"
Jul 13 15:21:56.498852 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.498826    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d816e2-bf78-49ad-a03d-7e0d8eda90f8" containerName="revoke-keys"
Jul 13 15:21:56.498943 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.498883    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ce3b554-436a-427b-a8a9-ce2972b817d8" containerName="maas-api"
Jul 13 15:21:56.498943 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.498889    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="42d816e2-bf78-49ad-a03d-7e0d8eda90f8" containerName="revoke-keys"
Jul 13 15:21:56.498943 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.498896    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="072e8b23-d873-453e-b913-d758ff232830" containerName="revoke-keys"
Jul 13 15:21:56.502967 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.502949    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:21:56.505838 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.505814    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f99515a5-serving-cert\""
Jul 13 15:21:56.505963 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.505890    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvc6k\""
Jul 13 15:21:56.506777 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.506762    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 15:21:56.507234 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.507218    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 15:21:56.515557 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.515530    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"]
Jul 13 15:21:56.555183 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.555147    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78b8ea2-0cea-4524-b324-e3ad00c42470-openshift-service-ca\") pod \"maas-api-e2e-derive-f99515a5-7799875886-gtbdq\" (UID: \"f78b8ea2-0cea-4524-b324-e3ad00c42470\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:21:56.555363 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.555214    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mbsq\" (UniqueName: \"kubernetes.io/projected/f78b8ea2-0cea-4524-b324-e3ad00c42470-kube-api-access-9mbsq\") pod \"maas-api-e2e-derive-f99515a5-7799875886-gtbdq\" (UID: \"f78b8ea2-0cea-4524-b324-e3ad00c42470\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:21:56.555363 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.555243    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78b8ea2-0cea-4524-b324-e3ad00c42470-maas-api-tls\") pod \"maas-api-e2e-derive-f99515a5-7799875886-gtbdq\" (UID: \"f78b8ea2-0cea-4524-b324-e3ad00c42470\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:21:56.655918 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.655878    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78b8ea2-0cea-4524-b324-e3ad00c42470-openshift-service-ca\") pod \"maas-api-e2e-derive-f99515a5-7799875886-gtbdq\" (UID: \"f78b8ea2-0cea-4524-b324-e3ad00c42470\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:21:56.656080 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.655969    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mbsq\" (UniqueName: \"kubernetes.io/projected/f78b8ea2-0cea-4524-b324-e3ad00c42470-kube-api-access-9mbsq\") pod \"maas-api-e2e-derive-f99515a5-7799875886-gtbdq\" (UID: \"f78b8ea2-0cea-4524-b324-e3ad00c42470\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:21:56.656080 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.656001    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78b8ea2-0cea-4524-b324-e3ad00c42470-maas-api-tls\") pod \"maas-api-e2e-derive-f99515a5-7799875886-gtbdq\" (UID: \"f78b8ea2-0cea-4524-b324-e3ad00c42470\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:21:56.656156 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:21:56.656121    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-serving-cert: secret "maas-api-e2e-derive-f99515a5-serving-cert" not found
Jul 13 15:21:56.656205 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:21:56.656195    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f78b8ea2-0cea-4524-b324-e3ad00c42470-maas-api-tls podName:f78b8ea2-0cea-4524-b324-e3ad00c42470 nodeName:}" failed. No retries permitted until 2026-07-13 15:21:57.156175201 +0000 UTC m=+1743.353328127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f78b8ea2-0cea-4524-b324-e3ad00c42470-maas-api-tls") pod "maas-api-e2e-derive-f99515a5-7799875886-gtbdq" (UID: "f78b8ea2-0cea-4524-b324-e3ad00c42470") : secret "maas-api-e2e-derive-f99515a5-serving-cert" not found
Jul 13 15:21:56.656499 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.656480    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78b8ea2-0cea-4524-b324-e3ad00c42470-openshift-service-ca\") pod \"maas-api-e2e-derive-f99515a5-7799875886-gtbdq\" (UID: \"f78b8ea2-0cea-4524-b324-e3ad00c42470\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:21:56.667256 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:56.667235    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mbsq\" (UniqueName: \"kubernetes.io/projected/f78b8ea2-0cea-4524-b324-e3ad00c42470-kube-api-access-9mbsq\") pod \"maas-api-e2e-derive-f99515a5-7799875886-gtbdq\" (UID: \"f78b8ea2-0cea-4524-b324-e3ad00c42470\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:21:57.160406 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:57.160371    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78b8ea2-0cea-4524-b324-e3ad00c42470-maas-api-tls\") pod \"maas-api-e2e-derive-f99515a5-7799875886-gtbdq\" (UID: \"f78b8ea2-0cea-4524-b324-e3ad00c42470\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:21:57.163018 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:57.162993    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78b8ea2-0cea-4524-b324-e3ad00c42470-maas-api-tls\") pod \"maas-api-e2e-derive-f99515a5-7799875886-gtbdq\" (UID: \"f78b8ea2-0cea-4524-b324-e3ad00c42470\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:21:57.413573 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:57.413480    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:21:57.552628 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:57.552600    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"]
Jul 13 15:21:57.554230 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:21:57.554199    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78b8ea2_0cea_4524_b324_e3ad00c42470.slice/crio-69e077e3502ae3fec06a0b8ba1974f6e05a3707930435397c085cb5d1c8b7298 WatchSource:0}: Error finding container 69e077e3502ae3fec06a0b8ba1974f6e05a3707930435397c085cb5d1c8b7298: Status 404 returned error can't find the container with id 69e077e3502ae3fec06a0b8ba1974f6e05a3707930435397c085cb5d1c8b7298
Jul 13 15:21:58.134998 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:58.134966    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq" event={"ID":"f78b8ea2-0cea-4524-b324-e3ad00c42470","Type":"ContainerStarted","Data":"8edcb9857f1a3cd4ca25680b1289a6d1b2ab35cb870e0244a1067343fe4350aa"}
Jul 13 15:21:58.135127 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:58.135021    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq" event={"ID":"f78b8ea2-0cea-4524-b324-e3ad00c42470","Type":"ContainerStarted","Data":"69e077e3502ae3fec06a0b8ba1974f6e05a3707930435397c085cb5d1c8b7298"}
Jul 13 15:21:58.135127 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:58.135087    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:21:58.171062 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:21:58.171010    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq" podStartSLOduration=1.6960297199999999 podStartE2EDuration="2.170993402s" totalImagesPullingTime="474.963682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:21:58.168863039 +0000 UTC m=+1744.366015987" watchObservedRunningTime="2026-07-13 15:21:58.170993402 +0000 UTC m=+1744.368146395"
Jul 13 15:22:04.146889 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:22:04.146857    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:22:54.333272 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:22:54.333155    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:22:54.340291 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:22:54.333980    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:23:01.135709 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.135657    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"]
Jul 13 15:23:01.136119 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.135930    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq" podUID="f78b8ea2-0cea-4524-b324-e3ad00c42470" containerName="maas-api" containerID="cri-o://8edcb9857f1a3cd4ca25680b1289a6d1b2ab35cb870e0244a1067343fe4350aa" gracePeriod=30
Jul 13 15:23:01.397016 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.396990    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:23:01.434928 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.434897    2564 generic.go:356] "Generic (PLEG): container finished" podID="f78b8ea2-0cea-4524-b324-e3ad00c42470" containerID="8edcb9857f1a3cd4ca25680b1289a6d1b2ab35cb870e0244a1067343fe4350aa" exitCode=0
Jul 13 15:23:01.435116 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.434972    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"
Jul 13 15:23:01.435116 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.434991    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq" event={"ID":"f78b8ea2-0cea-4524-b324-e3ad00c42470","Type":"ContainerDied","Data":"8edcb9857f1a3cd4ca25680b1289a6d1b2ab35cb870e0244a1067343fe4350aa"}
Jul 13 15:23:01.435116 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.435045    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq" event={"ID":"f78b8ea2-0cea-4524-b324-e3ad00c42470","Type":"ContainerDied","Data":"69e077e3502ae3fec06a0b8ba1974f6e05a3707930435397c085cb5d1c8b7298"}
Jul 13 15:23:01.435116 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.435067    2564 scope.go:117] "RemoveContainer" containerID="8edcb9857f1a3cd4ca25680b1289a6d1b2ab35cb870e0244a1067343fe4350aa"
Jul 13 15:23:01.447096 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.447077    2564 scope.go:117] "RemoveContainer" containerID="8edcb9857f1a3cd4ca25680b1289a6d1b2ab35cb870e0244a1067343fe4350aa"
Jul 13 15:23:01.447391 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:23:01.447348    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8edcb9857f1a3cd4ca25680b1289a6d1b2ab35cb870e0244a1067343fe4350aa\": container with ID starting with 8edcb9857f1a3cd4ca25680b1289a6d1b2ab35cb870e0244a1067343fe4350aa not found: ID does not exist" containerID="8edcb9857f1a3cd4ca25680b1289a6d1b2ab35cb870e0244a1067343fe4350aa"
Jul 13 15:23:01.447391 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.447378    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8edcb9857f1a3cd4ca25680b1289a6d1b2ab35cb870e0244a1067343fe4350aa"} err="failed to get container status \"8edcb9857f1a3cd4ca25680b1289a6d1b2ab35cb870e0244a1067343fe4350aa\": rpc error: code = NotFound desc = could not find container \"8edcb9857f1a3cd4ca25680b1289a6d1b2ab35cb870e0244a1067343fe4350aa\": container with ID starting with 8edcb9857f1a3cd4ca25680b1289a6d1b2ab35cb870e0244a1067343fe4350aa not found: ID does not exist"
Jul 13 15:23:01.532003 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.531963    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78b8ea2-0cea-4524-b324-e3ad00c42470-maas-api-tls\") pod \"f78b8ea2-0cea-4524-b324-e3ad00c42470\" (UID: \"f78b8ea2-0cea-4524-b324-e3ad00c42470\") "
Jul 13 15:23:01.532003 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.532011    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mbsq\" (UniqueName: \"kubernetes.io/projected/f78b8ea2-0cea-4524-b324-e3ad00c42470-kube-api-access-9mbsq\") pod \"f78b8ea2-0cea-4524-b324-e3ad00c42470\" (UID: \"f78b8ea2-0cea-4524-b324-e3ad00c42470\") "
Jul 13 15:23:01.532223 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.532097    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78b8ea2-0cea-4524-b324-e3ad00c42470-openshift-service-ca\") pod \"f78b8ea2-0cea-4524-b324-e3ad00c42470\" (UID: \"f78b8ea2-0cea-4524-b324-e3ad00c42470\") "
Jul 13 15:23:01.532494 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.532466    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78b8ea2-0cea-4524-b324-e3ad00c42470-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f78b8ea2-0cea-4524-b324-e3ad00c42470" (UID: "f78b8ea2-0cea-4524-b324-e3ad00c42470"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:23:01.534168 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.534145    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f78b8ea2-0cea-4524-b324-e3ad00c42470-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f78b8ea2-0cea-4524-b324-e3ad00c42470" (UID: "f78b8ea2-0cea-4524-b324-e3ad00c42470"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:23:01.534247 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.534237    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78b8ea2-0cea-4524-b324-e3ad00c42470-kube-api-access-9mbsq" (OuterVolumeSpecName: "kube-api-access-9mbsq") pod "f78b8ea2-0cea-4524-b324-e3ad00c42470" (UID: "f78b8ea2-0cea-4524-b324-e3ad00c42470"). InnerVolumeSpecName "kube-api-access-9mbsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:23:01.633013 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.632980    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78b8ea2-0cea-4524-b324-e3ad00c42470-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:23:01.633013 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.633007    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mbsq\" (UniqueName: \"kubernetes.io/projected/f78b8ea2-0cea-4524-b324-e3ad00c42470-kube-api-access-9mbsq\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:23:01.633013 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.633016    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78b8ea2-0cea-4524-b324-e3ad00c42470-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:23:01.766016 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.765984    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"]
Jul 13 15:23:01.782418 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:01.782387    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f99515a5-7799875886-gtbdq"]
Jul 13 15:23:02.318864 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:23:02.318836    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f78b8ea2-0cea-4524-b324-e3ad00c42470" path="/var/lib/kubelet/pods/f78b8ea2-0cea-4524-b324-e3ad00c42470/volumes"
Jul 13 15:24:57.580461 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.577731    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"]
Jul 13 15:24:57.580461 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.578592    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f78b8ea2-0cea-4524-b324-e3ad00c42470" containerName="maas-api"
Jul 13 15:24:57.580461 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.578612    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f78b8ea2-0cea-4524-b324-e3ad00c42470" containerName="maas-api"
Jul 13 15:24:57.580461 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.578815    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f78b8ea2-0cea-4524-b324-e3ad00c42470" containerName="maas-api"
Jul 13 15:24:57.583420 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.582672    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:24:57.585289 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.585263    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 15:24:57.585416 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.585298    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 15:24:57.585998 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.585981    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvc6k\""
Jul 13 15:24:57.586073 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.585996    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f969cfbe-serving-cert\""
Jul 13 15:24:57.590463 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.590348    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"]
Jul 13 15:24:57.623940 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.623913    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxpkf\" (UniqueName: \"kubernetes.io/projected/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-kube-api-access-rxpkf\") pod \"maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc\" (UID: \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:24:57.624100 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.623954    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-maas-api-tls\") pod \"maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc\" (UID: \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:24:57.624100 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.623987    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-openshift-service-ca\") pod \"maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc\" (UID: \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:24:57.724912 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.724879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-maas-api-tls\") pod \"maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc\" (UID: \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:24:57.725090 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.724929    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-openshift-service-ca\") pod \"maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc\" (UID: \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:24:57.725090 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.724968    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxpkf\" (UniqueName: \"kubernetes.io/projected/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-kube-api-access-rxpkf\") pod \"maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc\" (UID: \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:24:57.725090 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:24:57.725026    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-serving-cert: secret "maas-api-e2e-mt-f969cfbe-serving-cert" not found
Jul 13 15:24:57.725268 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:24:57.725126    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-maas-api-tls podName:d79cd5ab-217c-4704-baaa-2d06b0d7ad8e nodeName:}" failed. No retries permitted until 2026-07-13 15:24:58.225107383 +0000 UTC m=+1924.422260330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-maas-api-tls") pod "maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc" (UID: "d79cd5ab-217c-4704-baaa-2d06b0d7ad8e") : secret "maas-api-e2e-mt-f969cfbe-serving-cert" not found
Jul 13 15:24:57.725656 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.725633    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-openshift-service-ca\") pod \"maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc\" (UID: \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:24:57.734464 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:57.734442    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxpkf\" (UniqueName: \"kubernetes.io/projected/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-kube-api-access-rxpkf\") pod \"maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc\" (UID: \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:24:58.229275 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:58.229240    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-maas-api-tls\") pod \"maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc\" (UID: \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:24:58.231774 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:58.231751    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-maas-api-tls\") pod \"maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc\" (UID: \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:24:58.495197 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:58.495107    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:24:58.638971 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:58.638948    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"]
Jul 13 15:24:58.641079 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:24:58.641051    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79cd5ab_217c_4704_baaa_2d06b0d7ad8e.slice/crio-3243f2ca6e3a8762e51e9ab4bf29f72ea1e0ce0fd3a23b0ba51752c6cab45ef4 WatchSource:0}: Error finding container 3243f2ca6e3a8762e51e9ab4bf29f72ea1e0ce0fd3a23b0ba51752c6cab45ef4: Status 404 returned error can't find the container with id 3243f2ca6e3a8762e51e9ab4bf29f72ea1e0ce0fd3a23b0ba51752c6cab45ef4
Jul 13 15:24:58.642350 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:58.642332    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 15:24:58.971757 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:58.971718    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc" event={"ID":"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e","Type":"ContainerStarted","Data":"3243f2ca6e3a8762e51e9ab4bf29f72ea1e0ce0fd3a23b0ba51752c6cab45ef4"}
Jul 13 15:24:59.977952 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:59.977917    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc" event={"ID":"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e","Type":"ContainerStarted","Data":"407ac1945a1ce2b25ecd976c22f1aaafe8d8f19fae9c7e18ad7b242879f4c509"}
Jul 13 15:24:59.978452 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:59.978047    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:24:59.998042 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:24:59.997988    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc" podStartSLOduration=2.469282455 podStartE2EDuration="2.997972209s" totalImagesPullingTime="528.689754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:24:59.996280732 +0000 UTC m=+1926.193433679" watchObservedRunningTime="2026-07-13 15:24:59.997972209 +0000 UTC m=+1926.195125157"
Jul 13 15:25:05.989149 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:05.989116    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:25:09.764218 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:09.764178    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq"]
Jul 13 15:25:09.767908 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:09.767890    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq"
Jul 13 15:25:09.771222 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:09.771200    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\""
Jul 13 15:25:09.776253 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:09.776229    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq"]
Jul 13 15:25:09.828356 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:09.828323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18d6e559-6482-4a46-b864-f13115e1f85c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq\" (UID: \"18d6e559-6482-4a46-b864-f13115e1f85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq"
Jul 13 15:25:09.929314 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:09.929277    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18d6e559-6482-4a46-b864-f13115e1f85c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq\" (UID: \"18d6e559-6482-4a46-b864-f13115e1f85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq"
Jul 13 15:25:09.930005 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:09.929978    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18d6e559-6482-4a46-b864-f13115e1f85c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq\" (UID: \"18d6e559-6482-4a46-b864-f13115e1f85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq"
Jul 13 15:25:10.079497 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:10.079409    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq"
Jul 13 15:25:10.217045 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:10.217018    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq"]
Jul 13 15:25:10.219643 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:25:10.219597    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d6e559_6482_4a46_b864_f13115e1f85c.slice/crio-425a0c18878b9e67d448595b8f4bce9ca76b60dd77e2ce7d5bb8129046f182af WatchSource:0}: Error finding container 425a0c18878b9e67d448595b8f4bce9ca76b60dd77e2ce7d5bb8129046f182af: Status 404 returned error can't find the container with id 425a0c18878b9e67d448595b8f4bce9ca76b60dd77e2ce7d5bb8129046f182af
Jul 13 15:25:11.039320 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:11.039282    2564 generic.go:356] "Generic (PLEG): container finished" podID="18d6e559-6482-4a46-b864-f13115e1f85c" containerID="22fed50836b0b0982789b0f1cbaebf029965c8eb78d9e30468b9e24c4cad5915" exitCode=0
Jul 13 15:25:11.039691 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:11.039356    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq" event={"ID":"18d6e559-6482-4a46-b864-f13115e1f85c","Type":"ContainerDied","Data":"22fed50836b0b0982789b0f1cbaebf029965c8eb78d9e30468b9e24c4cad5915"}
Jul 13 15:25:11.039691 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:11.039396    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq" event={"ID":"18d6e559-6482-4a46-b864-f13115e1f85c","Type":"ContainerStarted","Data":"425a0c18878b9e67d448595b8f4bce9ca76b60dd77e2ce7d5bb8129046f182af"}
Jul 13 15:25:12.173428 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:12.173401    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq"
Jul 13 15:25:12.250973 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:12.250940    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18d6e559-6482-4a46-b864-f13115e1f85c-maas-api-service-ca\") pod \"18d6e559-6482-4a46-b864-f13115e1f85c\" (UID: \"18d6e559-6482-4a46-b864-f13115e1f85c\") "
Jul 13 15:25:12.251335 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:12.251309    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d6e559-6482-4a46-b864-f13115e1f85c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18d6e559-6482-4a46-b864-f13115e1f85c" (UID: "18d6e559-6482-4a46-b864-f13115e1f85c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:25:12.352539 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:12.352502    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18d6e559-6482-4a46-b864-f13115e1f85c-maas-api-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:25:13.049730 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:13.049677    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq" event={"ID":"18d6e559-6482-4a46-b864-f13115e1f85c","Type":"ContainerDied","Data":"425a0c18878b9e67d448595b8f4bce9ca76b60dd77e2ce7d5bb8129046f182af"}
Jul 13 15:25:13.049730 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:13.049723    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq"
Jul 13 15:25:13.049930 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:13.049748    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="425a0c18878b9e67d448595b8f4bce9ca76b60dd77e2ce7d5bb8129046f182af"
Jul 13 15:25:20.310019 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:20.309975    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"]
Jul 13 15:25:20.310372 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:20.310219    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc" podUID="d79cd5ab-217c-4704-baaa-2d06b0d7ad8e" containerName="maas-api" containerID="cri-o://407ac1945a1ce2b25ecd976c22f1aaafe8d8f19fae9c7e18ad7b242879f4c509" gracePeriod=30
Jul 13 15:25:20.563862 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:20.563799    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:25:20.623123 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:20.623090    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-openshift-service-ca\") pod \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\" (UID: \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\") "
Jul 13 15:25:20.623293 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:20.623176    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-maas-api-tls\") pod \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\" (UID: \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\") "
Jul 13 15:25:20.623293 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:20.623218    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxpkf\" (UniqueName: \"kubernetes.io/projected/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-kube-api-access-rxpkf\") pod \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\" (UID: \"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e\") "
Jul 13 15:25:20.623471 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:20.623446    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d79cd5ab-217c-4704-baaa-2d06b0d7ad8e" (UID: "d79cd5ab-217c-4704-baaa-2d06b0d7ad8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:25:20.625423 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:20.625391    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d79cd5ab-217c-4704-baaa-2d06b0d7ad8e" (UID: "d79cd5ab-217c-4704-baaa-2d06b0d7ad8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:25:20.625423 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:20.625412    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-kube-api-access-rxpkf" (OuterVolumeSpecName: "kube-api-access-rxpkf") pod "d79cd5ab-217c-4704-baaa-2d06b0d7ad8e" (UID: "d79cd5ab-217c-4704-baaa-2d06b0d7ad8e"). InnerVolumeSpecName "kube-api-access-rxpkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:25:20.724633 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:20.724600    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:25:20.724633 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:20.724626    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:25:20.724633 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:20.724635    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxpkf\" (UniqueName: \"kubernetes.io/projected/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e-kube-api-access-rxpkf\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:25:21.088758 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:21.088720    2564 generic.go:356] "Generic (PLEG): container finished" podID="d79cd5ab-217c-4704-baaa-2d06b0d7ad8e" containerID="407ac1945a1ce2b25ecd976c22f1aaafe8d8f19fae9c7e18ad7b242879f4c509" exitCode=0
Jul 13 15:25:21.088930 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:21.088772    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"
Jul 13 15:25:21.088930 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:21.088784    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc" event={"ID":"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e","Type":"ContainerDied","Data":"407ac1945a1ce2b25ecd976c22f1aaafe8d8f19fae9c7e18ad7b242879f4c509"}
Jul 13 15:25:21.088930 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:21.088868    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc" event={"ID":"d79cd5ab-217c-4704-baaa-2d06b0d7ad8e","Type":"ContainerDied","Data":"3243f2ca6e3a8762e51e9ab4bf29f72ea1e0ce0fd3a23b0ba51752c6cab45ef4"}
Jul 13 15:25:21.088930 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:21.088891    2564 scope.go:117] "RemoveContainer" containerID="407ac1945a1ce2b25ecd976c22f1aaafe8d8f19fae9c7e18ad7b242879f4c509"
Jul 13 15:25:21.098152 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:21.098136    2564 scope.go:117] "RemoveContainer" containerID="407ac1945a1ce2b25ecd976c22f1aaafe8d8f19fae9c7e18ad7b242879f4c509"
Jul 13 15:25:21.098404 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:25:21.098384    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407ac1945a1ce2b25ecd976c22f1aaafe8d8f19fae9c7e18ad7b242879f4c509\": container with ID starting with 407ac1945a1ce2b25ecd976c22f1aaafe8d8f19fae9c7e18ad7b242879f4c509 not found: ID does not exist" containerID="407ac1945a1ce2b25ecd976c22f1aaafe8d8f19fae9c7e18ad7b242879f4c509"
Jul 13 15:25:21.098470 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:21.098416    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407ac1945a1ce2b25ecd976c22f1aaafe8d8f19fae9c7e18ad7b242879f4c509"} err="failed to get container status \"407ac1945a1ce2b25ecd976c22f1aaafe8d8f19fae9c7e18ad7b242879f4c509\": rpc error: code = NotFound desc = could not find container \"407ac1945a1ce2b25ecd976c22f1aaafe8d8f19fae9c7e18ad7b242879f4c509\": container with ID starting with 407ac1945a1ce2b25ecd976c22f1aaafe8d8f19fae9c7e18ad7b242879f4c509 not found: ID does not exist"
Jul 13 15:25:21.111760 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:21.111735    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"]
Jul 13 15:25:21.116071 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:21.116051    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f969cfbe-85fb4c456c-vx5dc"]
Jul 13 15:25:22.316253 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:22.316221    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d79cd5ab-217c-4704-baaa-2d06b0d7ad8e" path="/var/lib/kubelet/pods/d79cd5ab-217c-4704-baaa-2d06b0d7ad8e/volumes"
Jul 13 15:25:58.910547 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.910513    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"]
Jul 13 15:25:58.911041 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.910926    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d79cd5ab-217c-4704-baaa-2d06b0d7ad8e" containerName="maas-api"
Jul 13 15:25:58.911041 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.910940    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d79cd5ab-217c-4704-baaa-2d06b0d7ad8e" containerName="maas-api"
Jul 13 15:25:58.911041 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.910957    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18d6e559-6482-4a46-b864-f13115e1f85c" containerName="revoke-keys"
Jul 13 15:25:58.911041 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.910963    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="18d6e559-6482-4a46-b864-f13115e1f85c" containerName="revoke-keys"
Jul 13 15:25:58.911041 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.911021    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="18d6e559-6482-4a46-b864-f13115e1f85c" containerName="revoke-keys"
Jul 13 15:25:58.911041 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.911033    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d79cd5ab-217c-4704-baaa-2d06b0d7ad8e" containerName="maas-api"
Jul 13 15:25:58.913882 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.913866    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:25:58.916752 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.916731    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fb1236df-serving-cert\""
Jul 13 15:25:58.916866 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.916816    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 15:25:58.917472 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.917459    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 15:25:58.917519 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.917462    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvc6k\""
Jul 13 15:25:58.925665 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.925644    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"]
Jul 13 15:25:58.974579 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.974538    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhp22\" (UniqueName: \"kubernetes.io/projected/640347ed-7311-49ab-adbe-2e4ec0db263e-kube-api-access-xhp22\") pod \"maas-api-e2e-mt-fb1236df-68668d8475-r549n\" (UID: \"640347ed-7311-49ab-adbe-2e4ec0db263e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:25:58.974781 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.974594    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/640347ed-7311-49ab-adbe-2e4ec0db263e-openshift-service-ca\") pod \"maas-api-e2e-mt-fb1236df-68668d8475-r549n\" (UID: \"640347ed-7311-49ab-adbe-2e4ec0db263e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:25:58.974781 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:58.974669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/640347ed-7311-49ab-adbe-2e4ec0db263e-maas-api-tls\") pod \"maas-api-e2e-mt-fb1236df-68668d8475-r549n\" (UID: \"640347ed-7311-49ab-adbe-2e4ec0db263e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:25:59.075613 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:59.075578    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhp22\" (UniqueName: \"kubernetes.io/projected/640347ed-7311-49ab-adbe-2e4ec0db263e-kube-api-access-xhp22\") pod \"maas-api-e2e-mt-fb1236df-68668d8475-r549n\" (UID: \"640347ed-7311-49ab-adbe-2e4ec0db263e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:25:59.075812 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:59.075640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/640347ed-7311-49ab-adbe-2e4ec0db263e-openshift-service-ca\") pod \"maas-api-e2e-mt-fb1236df-68668d8475-r549n\" (UID: \"640347ed-7311-49ab-adbe-2e4ec0db263e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:25:59.075812 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:59.075691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/640347ed-7311-49ab-adbe-2e4ec0db263e-maas-api-tls\") pod \"maas-api-e2e-mt-fb1236df-68668d8475-r549n\" (UID: \"640347ed-7311-49ab-adbe-2e4ec0db263e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:25:59.076363 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:59.076342    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/640347ed-7311-49ab-adbe-2e4ec0db263e-openshift-service-ca\") pod \"maas-api-e2e-mt-fb1236df-68668d8475-r549n\" (UID: \"640347ed-7311-49ab-adbe-2e4ec0db263e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:25:59.078258 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:59.078239    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/640347ed-7311-49ab-adbe-2e4ec0db263e-maas-api-tls\") pod \"maas-api-e2e-mt-fb1236df-68668d8475-r549n\" (UID: \"640347ed-7311-49ab-adbe-2e4ec0db263e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:25:59.085039 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:59.085016    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhp22\" (UniqueName: \"kubernetes.io/projected/640347ed-7311-49ab-adbe-2e4ec0db263e-kube-api-access-xhp22\") pod \"maas-api-e2e-mt-fb1236df-68668d8475-r549n\" (UID: \"640347ed-7311-49ab-adbe-2e4ec0db263e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:25:59.225555 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:59.225467    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:25:59.420427 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:25:59.420401    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"]
Jul 13 15:25:59.422872 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:25:59.422839    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640347ed_7311_49ab_adbe_2e4ec0db263e.slice/crio-e1abfed4ccf81437f0ef8f92632b85c6d371e58380d884debfc16eee33263663 WatchSource:0}: Error finding container e1abfed4ccf81437f0ef8f92632b85c6d371e58380d884debfc16eee33263663: Status 404 returned error can't find the container with id e1abfed4ccf81437f0ef8f92632b85c6d371e58380d884debfc16eee33263663
Jul 13 15:26:00.277681 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:00.277647    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n" event={"ID":"640347ed-7311-49ab-adbe-2e4ec0db263e","Type":"ContainerStarted","Data":"49984882b627ada360825e23d8905607e35d59731a88202a97ad50a022df2ce8"}
Jul 13 15:26:00.278120 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:00.277761    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n" event={"ID":"640347ed-7311-49ab-adbe-2e4ec0db263e","Type":"ContainerStarted","Data":"e1abfed4ccf81437f0ef8f92632b85c6d371e58380d884debfc16eee33263663"}
Jul 13 15:26:00.278120 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:00.277795    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:26:00.298137 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:00.298088    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n" podStartSLOduration=1.919759846 podStartE2EDuration="2.29807183s" totalImagesPullingTime="378.311984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:26:00.296180487 +0000 UTC m=+1986.493333435" watchObservedRunningTime="2026-07-13 15:26:00.29807183 +0000 UTC m=+1986.495224777"
Jul 13 15:26:06.288586 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:06.288549    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:26:24.484935 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.484899    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"]
Jul 13 15:26:24.488616 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.488591    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.491130 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.491104    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fb1236df\"/\"kube-root-ca.crt\""
Jul 13 15:26:24.491857 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.491831    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fb1236df\"/\"openshift-service-ca.crt\""
Jul 13 15:26:24.491857 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.491855    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fb1236df\"/\"default-dockercfg-t9mbl\""
Jul 13 15:26:24.492047 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.491842    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fb1236df\"/\"e2e-lifecycle-model-fb1236df-kserve-self-signed-certs\""
Jul 13 15:26:24.498369 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.498348    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"]
Jul 13 15:26:24.503648 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.503623    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng9zb\" (UniqueName: \"kubernetes.io/projected/0d884416-88a1-4aa4-a62d-4b0899773333-kube-api-access-ng9zb\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.503772 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.503680    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-tmp-dir\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.503772 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.503757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-dshm\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.503920 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.503789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-home\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.503920 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.503823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d884416-88a1-4aa4-a62d-4b0899773333-tls-certs\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.503920 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.503850    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-model-cache\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.605121 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.605077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng9zb\" (UniqueName: \"kubernetes.io/projected/0d884416-88a1-4aa4-a62d-4b0899773333-kube-api-access-ng9zb\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.605334 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.605130    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-tmp-dir\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.605334 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.605150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-dshm\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.605334 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.605169    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-home\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.605334 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.605201    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d884416-88a1-4aa4-a62d-4b0899773333-tls-certs\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.605334 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.605220    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-model-cache\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.605662 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.605641    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-model-cache\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.605764 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.605672    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-tmp-dir\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.605764 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.605734    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-home\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.607648 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.607626    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-dshm\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.607974 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.607955    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d884416-88a1-4aa4-a62d-4b0899773333-tls-certs\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.614574 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.614551    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng9zb\" (UniqueName: \"kubernetes.io/projected/0d884416-88a1-4aa4-a62d-4b0899773333-kube-api-access-ng9zb\") pod \"e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") " pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.800427 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.800334    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:24.941167 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:24.941139    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"]
Jul 13 15:26:24.943058 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:26:24.943025    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d884416_88a1_4aa4_a62d_4b0899773333.slice/crio-71778e6b26d04bd2cb01ca6d7a6447c1b3d8856743da3fddb1718671cff67a26 WatchSource:0}: Error finding container 71778e6b26d04bd2cb01ca6d7a6447c1b3d8856743da3fddb1718671cff67a26: Status 404 returned error can't find the container with id 71778e6b26d04bd2cb01ca6d7a6447c1b3d8856743da3fddb1718671cff67a26
Jul 13 15:26:25.401058 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:25.401022    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd" event={"ID":"0d884416-88a1-4aa4-a62d-4b0899773333","Type":"ContainerStarted","Data":"136aa51b1fac4675635487e8ea78e32400c9e2a31a7b7e2f061c04e9e3c5fc26"}
Jul 13 15:26:25.401255 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:25.401069    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd" event={"ID":"0d884416-88a1-4aa4-a62d-4b0899773333","Type":"ContainerStarted","Data":"71778e6b26d04bd2cb01ca6d7a6447c1b3d8856743da3fddb1718671cff67a26"}
Jul 13 15:26:25.428819 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:25.428766    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd" podStartSLOduration=1.428749839 podStartE2EDuration="1.428749839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:26:25.428442808 +0000 UTC m=+2011.625595755" watchObservedRunningTime="2026-07-13 15:26:25.428749839 +0000 UTC m=+2011.625902787"
Jul 13 15:26:34.800831 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:34.800796    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:34.801374 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:34.800875    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:34.813775 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:34.813745    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:35.460729 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:35.460685    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:26:54.416635 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:26:54.416604    2564 scope.go:117] "RemoveContainer" containerID="35c3cbd7e7d804ae148e8d26ad18619cd5cec1cebedeb39091817c8629f721fa"
Jul 13 15:27:02.595385 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:02.595345    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"]
Jul 13 15:27:02.595948 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:02.595603    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n" podUID="640347ed-7311-49ab-adbe-2e4ec0db263e" containerName="maas-api" containerID="cri-o://49984882b627ada360825e23d8905607e35d59731a88202a97ad50a022df2ce8" gracePeriod=30
Jul 13 15:27:02.847580 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:02.847520    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:27:02.950107 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:02.950073    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhp22\" (UniqueName: \"kubernetes.io/projected/640347ed-7311-49ab-adbe-2e4ec0db263e-kube-api-access-xhp22\") pod \"640347ed-7311-49ab-adbe-2e4ec0db263e\" (UID: \"640347ed-7311-49ab-adbe-2e4ec0db263e\") "
Jul 13 15:27:02.950310 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:02.950136    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/640347ed-7311-49ab-adbe-2e4ec0db263e-openshift-service-ca\") pod \"640347ed-7311-49ab-adbe-2e4ec0db263e\" (UID: \"640347ed-7311-49ab-adbe-2e4ec0db263e\") "
Jul 13 15:27:02.950310 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:02.950221    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/640347ed-7311-49ab-adbe-2e4ec0db263e-maas-api-tls\") pod \"640347ed-7311-49ab-adbe-2e4ec0db263e\" (UID: \"640347ed-7311-49ab-adbe-2e4ec0db263e\") "
Jul 13 15:27:02.950574 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:02.950551    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/640347ed-7311-49ab-adbe-2e4ec0db263e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "640347ed-7311-49ab-adbe-2e4ec0db263e" (UID: "640347ed-7311-49ab-adbe-2e4ec0db263e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:27:02.952408 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:02.952383    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/640347ed-7311-49ab-adbe-2e4ec0db263e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "640347ed-7311-49ab-adbe-2e4ec0db263e" (UID: "640347ed-7311-49ab-adbe-2e4ec0db263e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:27:02.952497 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:02.952464    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/640347ed-7311-49ab-adbe-2e4ec0db263e-kube-api-access-xhp22" (OuterVolumeSpecName: "kube-api-access-xhp22") pod "640347ed-7311-49ab-adbe-2e4ec0db263e" (UID: "640347ed-7311-49ab-adbe-2e4ec0db263e"). InnerVolumeSpecName "kube-api-access-xhp22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:27:03.051565 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:03.051528    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/640347ed-7311-49ab-adbe-2e4ec0db263e-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:27:03.051565 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:03.051558    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/640347ed-7311-49ab-adbe-2e4ec0db263e-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:27:03.051565 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:03.051567    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhp22\" (UniqueName: \"kubernetes.io/projected/640347ed-7311-49ab-adbe-2e4ec0db263e-kube-api-access-xhp22\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:27:03.601154 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:03.601117    2564 generic.go:356] "Generic (PLEG): container finished" podID="640347ed-7311-49ab-adbe-2e4ec0db263e" containerID="49984882b627ada360825e23d8905607e35d59731a88202a97ad50a022df2ce8" exitCode=0
Jul 13 15:27:03.601574 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:03.601160    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"
Jul 13 15:27:03.601574 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:03.601170    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n" event={"ID":"640347ed-7311-49ab-adbe-2e4ec0db263e","Type":"ContainerDied","Data":"49984882b627ada360825e23d8905607e35d59731a88202a97ad50a022df2ce8"}
Jul 13 15:27:03.601574 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:03.601229    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n" event={"ID":"640347ed-7311-49ab-adbe-2e4ec0db263e","Type":"ContainerDied","Data":"e1abfed4ccf81437f0ef8f92632b85c6d371e58380d884debfc16eee33263663"}
Jul 13 15:27:03.601574 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:03.601253    2564 scope.go:117] "RemoveContainer" containerID="49984882b627ada360825e23d8905607e35d59731a88202a97ad50a022df2ce8"
Jul 13 15:27:03.610597 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:03.610582    2564 scope.go:117] "RemoveContainer" containerID="49984882b627ada360825e23d8905607e35d59731a88202a97ad50a022df2ce8"
Jul 13 15:27:03.610857 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:27:03.610835    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49984882b627ada360825e23d8905607e35d59731a88202a97ad50a022df2ce8\": container with ID starting with 49984882b627ada360825e23d8905607e35d59731a88202a97ad50a022df2ce8 not found: ID does not exist" containerID="49984882b627ada360825e23d8905607e35d59731a88202a97ad50a022df2ce8"
Jul 13 15:27:03.610909 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:03.610866    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49984882b627ada360825e23d8905607e35d59731a88202a97ad50a022df2ce8"} err="failed to get container status \"49984882b627ada360825e23d8905607e35d59731a88202a97ad50a022df2ce8\": rpc error: code = NotFound desc = could not find container \"49984882b627ada360825e23d8905607e35d59731a88202a97ad50a022df2ce8\": container with ID starting with 49984882b627ada360825e23d8905607e35d59731a88202a97ad50a022df2ce8 not found: ID does not exist"
Jul 13 15:27:03.623811 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:03.623787    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"]
Jul 13 15:27:03.627585 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:03.627565    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb1236df-68668d8475-r549n"]
Jul 13 15:27:04.316347 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:04.316273    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="640347ed-7311-49ab-adbe-2e4ec0db263e" path="/var/lib/kubelet/pods/640347ed-7311-49ab-adbe-2e4ec0db263e/volumes"
Jul 13 15:27:12.157494 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:12.157460    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"]
Jul 13 15:27:27.178752 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.178632    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd" podUID="0d884416-88a1-4aa4-a62d-4b0899773333" containerName="main" containerID="cri-o://136aa51b1fac4675635487e8ea78e32400c9e2a31a7b7e2f061c04e9e3c5fc26" gracePeriod=45
Jul 13 15:27:27.430870 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.430812    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:27:27.559632 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.559597    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-dshm\") pod \"0d884416-88a1-4aa4-a62d-4b0899773333\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") "
Jul 13 15:27:27.559632 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.559634    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-model-cache\") pod \"0d884416-88a1-4aa4-a62d-4b0899773333\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") "
Jul 13 15:27:27.559874 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.559680    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-tmp-dir\") pod \"0d884416-88a1-4aa4-a62d-4b0899773333\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") "
Jul 13 15:27:27.559874 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.559749    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-home\") pod \"0d884416-88a1-4aa4-a62d-4b0899773333\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") "
Jul 13 15:27:27.559874 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.559790    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng9zb\" (UniqueName: \"kubernetes.io/projected/0d884416-88a1-4aa4-a62d-4b0899773333-kube-api-access-ng9zb\") pod \"0d884416-88a1-4aa4-a62d-4b0899773333\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") "
Jul 13 15:27:27.559874 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.559818    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d884416-88a1-4aa4-a62d-4b0899773333-tls-certs\") pod \"0d884416-88a1-4aa4-a62d-4b0899773333\" (UID: \"0d884416-88a1-4aa4-a62d-4b0899773333\") "
Jul 13 15:27:27.560073 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.559928    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d884416-88a1-4aa4-a62d-4b0899773333" (UID: "0d884416-88a1-4aa4-a62d-4b0899773333"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:27:27.560073 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.560036    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d884416-88a1-4aa4-a62d-4b0899773333" (UID: "0d884416-88a1-4aa4-a62d-4b0899773333"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:27:27.560073 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.560048    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-home" (OuterVolumeSpecName: "home") pod "0d884416-88a1-4aa4-a62d-4b0899773333" (UID: "0d884416-88a1-4aa4-a62d-4b0899773333"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:27:27.560224 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.560163    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:27:27.560224 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.560176    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:27:27.560224 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.560192    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:27:27.562079 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.562045    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d884416-88a1-4aa4-a62d-4b0899773333-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d884416-88a1-4aa4-a62d-4b0899773333" (UID: "0d884416-88a1-4aa4-a62d-4b0899773333"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:27:27.562079 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.562070    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-dshm" (OuterVolumeSpecName: "dshm") pod "0d884416-88a1-4aa4-a62d-4b0899773333" (UID: "0d884416-88a1-4aa4-a62d-4b0899773333"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:27:27.562215 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.562190    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d884416-88a1-4aa4-a62d-4b0899773333-kube-api-access-ng9zb" (OuterVolumeSpecName: "kube-api-access-ng9zb") pod "0d884416-88a1-4aa4-a62d-4b0899773333" (UID: "0d884416-88a1-4aa4-a62d-4b0899773333"). InnerVolumeSpecName "kube-api-access-ng9zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:27:27.661065 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.661035    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d884416-88a1-4aa4-a62d-4b0899773333-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:27:27.661065 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.661059    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng9zb\" (UniqueName: \"kubernetes.io/projected/0d884416-88a1-4aa4-a62d-4b0899773333-kube-api-access-ng9zb\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:27:27.661065 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.661068    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d884416-88a1-4aa4-a62d-4b0899773333-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:27:27.714855 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.714780    2564 generic.go:356] "Generic (PLEG): container finished" podID="0d884416-88a1-4aa4-a62d-4b0899773333" containerID="136aa51b1fac4675635487e8ea78e32400c9e2a31a7b7e2f061c04e9e3c5fc26" exitCode=0
Jul 13 15:27:27.714855 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.714817    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"
Jul 13 15:27:27.715043 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.714861    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd" event={"ID":"0d884416-88a1-4aa4-a62d-4b0899773333","Type":"ContainerDied","Data":"136aa51b1fac4675635487e8ea78e32400c9e2a31a7b7e2f061c04e9e3c5fc26"}
Jul 13 15:27:27.715043 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.714903    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd" event={"ID":"0d884416-88a1-4aa4-a62d-4b0899773333","Type":"ContainerDied","Data":"71778e6b26d04bd2cb01ca6d7a6447c1b3d8856743da3fddb1718671cff67a26"}
Jul 13 15:27:27.715043 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.714923    2564 scope.go:117] "RemoveContainer" containerID="136aa51b1fac4675635487e8ea78e32400c9e2a31a7b7e2f061c04e9e3c5fc26"
Jul 13 15:27:27.724205 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.724189    2564 scope.go:117] "RemoveContainer" containerID="136aa51b1fac4675635487e8ea78e32400c9e2a31a7b7e2f061c04e9e3c5fc26"
Jul 13 15:27:27.724465 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:27:27.724447    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136aa51b1fac4675635487e8ea78e32400c9e2a31a7b7e2f061c04e9e3c5fc26\": container with ID starting with 136aa51b1fac4675635487e8ea78e32400c9e2a31a7b7e2f061c04e9e3c5fc26 not found: ID does not exist" containerID="136aa51b1fac4675635487e8ea78e32400c9e2a31a7b7e2f061c04e9e3c5fc26"
Jul 13 15:27:27.724523 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.724473    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136aa51b1fac4675635487e8ea78e32400c9e2a31a7b7e2f061c04e9e3c5fc26"} err="failed to get container status \"136aa51b1fac4675635487e8ea78e32400c9e2a31a7b7e2f061c04e9e3c5fc26\": rpc error: code = NotFound desc = could not find container \"136aa51b1fac4675635487e8ea78e32400c9e2a31a7b7e2f061c04e9e3c5fc26\": container with ID starting with 136aa51b1fac4675635487e8ea78e32400c9e2a31a7b7e2f061c04e9e3c5fc26 not found: ID does not exist"
Jul 13 15:27:27.744841 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.744811    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"]
Jul 13 15:27:27.751258 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:27.751234    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-fb1236df/e2e-lifecycle-model-fb1236df-kserve-749fc8bd4c-7jnkd"]
Jul 13 15:27:28.317796 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:28.317764    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d884416-88a1-4aa4-a62d-4b0899773333" path="/var/lib/kubelet/pods/0d884416-88a1-4aa4-a62d-4b0899773333/volumes"
Jul 13 15:27:54.361441 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:54.361337    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:27:54.364066 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:27:54.363123    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:28:57.888297 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.888215    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"]
Jul 13 15:28:57.888757 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.888559    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d884416-88a1-4aa4-a62d-4b0899773333" containerName="main"
Jul 13 15:28:57.888757 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.888570    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d884416-88a1-4aa4-a62d-4b0899773333" containerName="main"
Jul 13 15:28:57.888757 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.888586    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="640347ed-7311-49ab-adbe-2e4ec0db263e" containerName="maas-api"
Jul 13 15:28:57.888757 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.888591    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="640347ed-7311-49ab-adbe-2e4ec0db263e" containerName="maas-api"
Jul 13 15:28:57.888757 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.888647    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="640347ed-7311-49ab-adbe-2e4ec0db263e" containerName="maas-api"
Jul 13 15:28:57.888757 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.888655    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d884416-88a1-4aa4-a62d-4b0899773333" containerName="main"
Jul 13 15:28:57.891682 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.891663    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:28:57.894280 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.894262    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvc6k\""
Jul 13 15:28:57.894390 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.894277    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-15d63b-serving-cert\""
Jul 13 15:28:57.894438 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.894409    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 15:28:57.895067 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.895053    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 15:28:57.904215 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.904192    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"]
Jul 13 15:28:57.953923 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.953891    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khlp5\" (UniqueName: \"kubernetes.io/projected/418e94c5-e3fd-4e6e-9826-456da5543121-kube-api-access-khlp5\") pod \"maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm\" (UID: \"418e94c5-e3fd-4e6e-9826-456da5543121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:28:57.954102 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.953980    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/418e94c5-e3fd-4e6e-9826-456da5543121-openshift-service-ca\") pod \"maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm\" (UID: \"418e94c5-e3fd-4e6e-9826-456da5543121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:28:57.954102 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:57.954034    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/418e94c5-e3fd-4e6e-9826-456da5543121-maas-api-tls\") pod \"maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm\" (UID: \"418e94c5-e3fd-4e6e-9826-456da5543121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:28:58.054949 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:58.054910    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khlp5\" (UniqueName: \"kubernetes.io/projected/418e94c5-e3fd-4e6e-9826-456da5543121-kube-api-access-khlp5\") pod \"maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm\" (UID: \"418e94c5-e3fd-4e6e-9826-456da5543121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:28:58.055122 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:58.054964    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/418e94c5-e3fd-4e6e-9826-456da5543121-openshift-service-ca\") pod \"maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm\" (UID: \"418e94c5-e3fd-4e6e-9826-456da5543121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:28:58.055122 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:58.054994    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/418e94c5-e3fd-4e6e-9826-456da5543121-maas-api-tls\") pod \"maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm\" (UID: \"418e94c5-e3fd-4e6e-9826-456da5543121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:28:58.055198 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:28:58.055147    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-serving-cert: secret "maas-api-e2e-inf-a-15d63b-serving-cert" not found
Jul 13 15:28:58.055231 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:28:58.055221    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/418e94c5-e3fd-4e6e-9826-456da5543121-maas-api-tls podName:418e94c5-e3fd-4e6e-9826-456da5543121 nodeName:}" failed. No retries permitted until 2026-07-13 15:28:58.555203234 +0000 UTC m=+2164.752356160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/418e94c5-e3fd-4e6e-9826-456da5543121-maas-api-tls") pod "maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm" (UID: "418e94c5-e3fd-4e6e-9826-456da5543121") : secret "maas-api-e2e-inf-a-15d63b-serving-cert" not found
Jul 13 15:28:58.055608 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:58.055590    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/418e94c5-e3fd-4e6e-9826-456da5543121-openshift-service-ca\") pod \"maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm\" (UID: \"418e94c5-e3fd-4e6e-9826-456da5543121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:28:58.075935 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:58.075902    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khlp5\" (UniqueName: \"kubernetes.io/projected/418e94c5-e3fd-4e6e-9826-456da5543121-kube-api-access-khlp5\") pod \"maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm\" (UID: \"418e94c5-e3fd-4e6e-9826-456da5543121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:28:58.559627 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:58.559587    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/418e94c5-e3fd-4e6e-9826-456da5543121-maas-api-tls\") pod \"maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm\" (UID: \"418e94c5-e3fd-4e6e-9826-456da5543121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:28:58.562138 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:58.562108    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/418e94c5-e3fd-4e6e-9826-456da5543121-maas-api-tls\") pod \"maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm\" (UID: \"418e94c5-e3fd-4e6e-9826-456da5543121\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:28:58.802146 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:58.802103    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:28:58.939984 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:58.939903    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"]
Jul 13 15:28:58.942248 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:28:58.942215    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418e94c5_e3fd_4e6e_9826_456da5543121.slice/crio-92932019ea4beac13cf24b55efb308cc47c2034a591c427d6388b7db133e2e1d WatchSource:0}: Error finding container 92932019ea4beac13cf24b55efb308cc47c2034a591c427d6388b7db133e2e1d: Status 404 returned error can't find the container with id 92932019ea4beac13cf24b55efb308cc47c2034a591c427d6388b7db133e2e1d
Jul 13 15:28:59.144497 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:28:59.144468    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm" event={"ID":"418e94c5-e3fd-4e6e-9826-456da5543121","Type":"ContainerStarted","Data":"92932019ea4beac13cf24b55efb308cc47c2034a591c427d6388b7db133e2e1d"}
Jul 13 15:29:00.150596 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:00.150562    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm" event={"ID":"418e94c5-e3fd-4e6e-9826-456da5543121","Type":"ContainerStarted","Data":"8a7284c711847e031f0228f91927dbad499fbd3a0488142bd1e006a5564bbb6f"}
Jul 13 15:29:00.151020 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:00.150721    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:29:00.168849 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:00.168799    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm" podStartSLOduration=2.708670642 podStartE2EDuration="3.168781663s" totalImagesPullingTime="460.111021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:29:00.16728533 +0000 UTC m=+2166.364438280" watchObservedRunningTime="2026-07-13 15:29:00.168781663 +0000 UTC m=+2166.365934612"
Jul 13 15:29:06.161086 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.161053    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:29:06.179633 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.179599    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"]
Jul 13 15:29:06.183846 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.183819    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:29:06.185930 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.185907    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f68651-serving-cert\""
Jul 13 15:29:06.195453 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.195415    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"]
Jul 13 15:29:06.227786 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.227753    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c32263-2f6b-4ec7-b070-6dd409049589-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j\" (UID: \"92c32263-2f6b-4ec7-b070-6dd409049589\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:29:06.227786 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.227798    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c32263-2f6b-4ec7-b070-6dd409049589-maas-api-tls\") pod \"maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j\" (UID: \"92c32263-2f6b-4ec7-b070-6dd409049589\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:29:06.228082 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.228053    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7nk4\" (UniqueName: \"kubernetes.io/projected/92c32263-2f6b-4ec7-b070-6dd409049589-kube-api-access-h7nk4\") pod \"maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j\" (UID: \"92c32263-2f6b-4ec7-b070-6dd409049589\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:29:06.328836 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.328807    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7nk4\" (UniqueName: \"kubernetes.io/projected/92c32263-2f6b-4ec7-b070-6dd409049589-kube-api-access-h7nk4\") pod \"maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j\" (UID: \"92c32263-2f6b-4ec7-b070-6dd409049589\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:29:06.329027 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.328860    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c32263-2f6b-4ec7-b070-6dd409049589-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j\" (UID: \"92c32263-2f6b-4ec7-b070-6dd409049589\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:29:06.329027 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.328992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c32263-2f6b-4ec7-b070-6dd409049589-maas-api-tls\") pod \"maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j\" (UID: \"92c32263-2f6b-4ec7-b070-6dd409049589\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:29:06.329414 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.329393    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c32263-2f6b-4ec7-b070-6dd409049589-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j\" (UID: \"92c32263-2f6b-4ec7-b070-6dd409049589\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:29:06.331616 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.331589    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c32263-2f6b-4ec7-b070-6dd409049589-maas-api-tls\") pod \"maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j\" (UID: \"92c32263-2f6b-4ec7-b070-6dd409049589\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:29:06.336810 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.336789    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7nk4\" (UniqueName: \"kubernetes.io/projected/92c32263-2f6b-4ec7-b070-6dd409049589-kube-api-access-h7nk4\") pod \"maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j\" (UID: \"92c32263-2f6b-4ec7-b070-6dd409049589\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:29:06.503295 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.503206    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:29:06.636052 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:06.636027    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"]
Jul 13 15:29:06.638746 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:29:06.638720    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c32263_2f6b_4ec7_b070_6dd409049589.slice/crio-76af43a61688ad4a8a8c46e4c9f20f41916f5ba0b6a34d9c3c408fdfb5173e8a WatchSource:0}: Error finding container 76af43a61688ad4a8a8c46e4c9f20f41916f5ba0b6a34d9c3c408fdfb5173e8a: Status 404 returned error can't find the container with id 76af43a61688ad4a8a8c46e4c9f20f41916f5ba0b6a34d9c3c408fdfb5173e8a
Jul 13 15:29:07.184886 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:07.184845    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j" event={"ID":"92c32263-2f6b-4ec7-b070-6dd409049589","Type":"ContainerStarted","Data":"76af43a61688ad4a8a8c46e4c9f20f41916f5ba0b6a34d9c3c408fdfb5173e8a"}
Jul 13 15:29:09.197097 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:09.197062    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j" event={"ID":"92c32263-2f6b-4ec7-b070-6dd409049589","Type":"ContainerStarted","Data":"025dc6c218565a3e1d1e0aa50654361b27076d76c5cfa0ccc50bf2f6b432bdf3"}
Jul 13 15:29:09.197486 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:09.197119    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:29:09.215231 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:09.215157    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j" podStartSLOduration=1.719751918 podStartE2EDuration="3.215135821s" totalImagesPullingTime="1.495383903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:29:09.214531943 +0000 UTC m=+2175.411684902" watchObservedRunningTime="2026-07-13 15:29:09.215135821 +0000 UTC m=+2175.412288765"
Jul 13 15:29:15.207665 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:15.207630    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:29:17.192926 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.192895    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"]
Jul 13 15:29:17.196534 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.196513    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.200507 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.200199    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-15d63b\"/\"kube-root-ca.crt\""
Jul 13 15:29:17.200507 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.200218    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-15d63b\"/\"default-dockercfg-kdwpm\""
Jul 13 15:29:17.200507 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.200199    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-15d63b\"/\"openshift-service-ca.crt\""
Jul 13 15:29:17.200507 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.200199    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-15d63b\"/\"test-model-15d63b-kserve-self-signed-certs\""
Jul 13 15:29:17.209135 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.209106    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"]
Jul 13 15:29:17.326895 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.326863    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-dshm\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.326895 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.326897    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/061ff01e-8445-458c-ba86-5e5a134259c6-tls-certs\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.327104 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.326933    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-tmp-dir\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.327104 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.327009    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-model-cache\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.327171 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.327102    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-home\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.327171 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.327150    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kmwh\" (UniqueName: \"kubernetes.io/projected/061ff01e-8445-458c-ba86-5e5a134259c6-kube-api-access-5kmwh\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.427846 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.427806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-dshm\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.427846 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.427841    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/061ff01e-8445-458c-ba86-5e5a134259c6-tls-certs\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.428064 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.427882    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-tmp-dir\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.428110 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.428082    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-model-cache\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.428201 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.428175    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-home\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.428269 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.428248    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-tmp-dir\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.428327 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.428258    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kmwh\" (UniqueName: \"kubernetes.io/projected/061ff01e-8445-458c-ba86-5e5a134259c6-kube-api-access-5kmwh\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.428383 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.428342    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-model-cache\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.428557 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.428533    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-home\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.430540 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.430519    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-dshm\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.430678 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.430661    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/061ff01e-8445-458c-ba86-5e5a134259c6-tls-certs\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.437011 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.436978    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kmwh\" (UniqueName: \"kubernetes.io/projected/061ff01e-8445-458c-ba86-5e5a134259c6-kube-api-access-5kmwh\") pod \"test-model-15d63b-kserve-59d9685d67-z4rj4\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") " pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.509802 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.509688    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:17.649342 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:17.646977    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"]
Jul 13 15:29:17.650794 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:29:17.650737    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod061ff01e_8445_458c_ba86_5e5a134259c6.slice/crio-c011dd1f97ebd29d3e4874160705566e0322e20713dbd0e6168986793587d513 WatchSource:0}: Error finding container c011dd1f97ebd29d3e4874160705566e0322e20713dbd0e6168986793587d513: Status 404 returned error can't find the container with id c011dd1f97ebd29d3e4874160705566e0322e20713dbd0e6168986793587d513
Jul 13 15:29:18.242071 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:18.242035    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4" event={"ID":"061ff01e-8445-458c-ba86-5e5a134259c6","Type":"ContainerStarted","Data":"57054b7e01bead983d28d96b0e13ad7b1a53122a1cc72671780a8cee8e3dca30"}
Jul 13 15:29:18.242449 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:18.242079    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4" event={"ID":"061ff01e-8445-458c-ba86-5e5a134259c6","Type":"ContainerStarted","Data":"c011dd1f97ebd29d3e4874160705566e0322e20713dbd0e6168986793587d513"}
Jul 13 15:29:18.258658 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:18.258606    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4" podStartSLOduration=1.258586248 podStartE2EDuration="1.258586248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:29:18.258102265 +0000 UTC m=+2184.455255226" watchObservedRunningTime="2026-07-13 15:29:18.258586248 +0000 UTC m=+2184.455739197"
Jul 13 15:29:27.510188 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:27.510147    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:27.510870 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:27.510210    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:27.523629 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:27.523604    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:28.299551 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:28.299523    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:29:42.775364 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.775328    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"]
Jul 13 15:29:42.778888 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.778871    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.781429 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.781404    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f68651\"/\"kube-root-ca.crt\""
Jul 13 15:29:42.782361 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.782342    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f68651\"/\"openshift-service-ca.crt\""
Jul 13 15:29:42.782361 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.782355    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f68651\"/\"test-model-f68651-kserve-self-signed-certs\""
Jul 13 15:29:42.782510 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.782360    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f68651\"/\"default-dockercfg-rsmnz\""
Jul 13 15:29:42.788051 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.788027    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"]
Jul 13 15:29:42.859537 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.859492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-model-cache\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.859537 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.859544    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-home\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.859802 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.859561    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-dshm\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.859802 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.859580    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-tmp-dir\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.859802 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.859665    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-tls-certs\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.859802 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.859722    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkhbj\" (UniqueName: \"kubernetes.io/projected/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-kube-api-access-mkhbj\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.960363 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.960327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-tls-certs\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.960363 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.960367    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkhbj\" (UniqueName: \"kubernetes.io/projected/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-kube-api-access-mkhbj\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.960587 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.960423    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-model-cache\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.960587 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.960455    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-home\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.960587 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.960471    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-dshm\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.960587 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.960509    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-tmp-dir\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.961005 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.960980    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-home\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.961082 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.961015    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-model-cache\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.961082 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.961040    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-tmp-dir\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.962898 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.962879    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-dshm\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.963039 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.963022    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-tls-certs\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:42.969156 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:42.969132    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkhbj\" (UniqueName: \"kubernetes.io/projected/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-kube-api-access-mkhbj\") pod \"test-model-f68651-kserve-6c7577f89-9pqw6\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") " pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:43.089808 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:43.089686    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:43.226650 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:43.226626    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"]
Jul 13 15:29:43.228672 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:29:43.228643    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc83eec51_9c3a_4355_a3e0_aa34087a8fbb.slice/crio-48ac5c3673806a40b7e733e1d567955d0761497d3897c755a41a9237003ea138 WatchSource:0}: Error finding container 48ac5c3673806a40b7e733e1d567955d0761497d3897c755a41a9237003ea138: Status 404 returned error can't find the container with id 48ac5c3673806a40b7e733e1d567955d0761497d3897c755a41a9237003ea138
Jul 13 15:29:43.363372 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:43.363269    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6" event={"ID":"c83eec51-9c3a-4355-a3e0-aa34087a8fbb","Type":"ContainerStarted","Data":"c855c125cb12ce18a2198efab91daddda95627fbeda44fe52c43301ff538054f"}
Jul 13 15:29:43.363372 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:43.363328    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6" event={"ID":"c83eec51-9c3a-4355-a3e0-aa34087a8fbb","Type":"ContainerStarted","Data":"48ac5c3673806a40b7e733e1d567955d0761497d3897c755a41a9237003ea138"}
Jul 13 15:29:43.380353 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:43.380298    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6" podStartSLOduration=1.380277397 podStartE2EDuration="1.380277397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:29:43.37898001 +0000 UTC m=+2209.576132958" watchObservedRunningTime="2026-07-13 15:29:43.380277397 +0000 UTC m=+2209.577430344"
Jul 13 15:29:53.090129 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:53.090085    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:53.090545 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:53.090168    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:53.102970 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:53.102941    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:29:53.422806 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:29:53.422776    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:30:00.675434 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:00.675401    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"]
Jul 13 15:30:02.017855 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:02.017819    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"]
Jul 13 15:30:12.622228 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:12.622194    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"]
Jul 13 15:30:12.622828 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:12.622421    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm" podUID="418e94c5-e3fd-4e6e-9826-456da5543121" containerName="maas-api" containerID="cri-o://8a7284c711847e031f0228f91927dbad499fbd3a0488142bd1e006a5564bbb6f" gracePeriod=30
Jul 13 15:30:12.872543 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:12.872485    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:30:13.045973 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.045938    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khlp5\" (UniqueName: \"kubernetes.io/projected/418e94c5-e3fd-4e6e-9826-456da5543121-kube-api-access-khlp5\") pod \"418e94c5-e3fd-4e6e-9826-456da5543121\" (UID: \"418e94c5-e3fd-4e6e-9826-456da5543121\") "
Jul 13 15:30:13.046283 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.046030    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/418e94c5-e3fd-4e6e-9826-456da5543121-maas-api-tls\") pod \"418e94c5-e3fd-4e6e-9826-456da5543121\" (UID: \"418e94c5-e3fd-4e6e-9826-456da5543121\") "
Jul 13 15:30:13.046283 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.046116    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/418e94c5-e3fd-4e6e-9826-456da5543121-openshift-service-ca\") pod \"418e94c5-e3fd-4e6e-9826-456da5543121\" (UID: \"418e94c5-e3fd-4e6e-9826-456da5543121\") "
Jul 13 15:30:13.046456 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.046434    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418e94c5-e3fd-4e6e-9826-456da5543121-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "418e94c5-e3fd-4e6e-9826-456da5543121" (UID: "418e94c5-e3fd-4e6e-9826-456da5543121"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:30:13.048366 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.048338    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418e94c5-e3fd-4e6e-9826-456da5543121-kube-api-access-khlp5" (OuterVolumeSpecName: "kube-api-access-khlp5") pod "418e94c5-e3fd-4e6e-9826-456da5543121" (UID: "418e94c5-e3fd-4e6e-9826-456da5543121"). InnerVolumeSpecName "kube-api-access-khlp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:30:13.048465 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.048379    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/418e94c5-e3fd-4e6e-9826-456da5543121-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "418e94c5-e3fd-4e6e-9826-456da5543121" (UID: "418e94c5-e3fd-4e6e-9826-456da5543121"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:30:13.147459 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.147434    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khlp5\" (UniqueName: \"kubernetes.io/projected/418e94c5-e3fd-4e6e-9826-456da5543121-kube-api-access-khlp5\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:13.147607 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.147462    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/418e94c5-e3fd-4e6e-9826-456da5543121-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:13.147607 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.147471    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/418e94c5-e3fd-4e6e-9826-456da5543121-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:13.509102 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.509007    2564 generic.go:356] "Generic (PLEG): container finished" podID="418e94c5-e3fd-4e6e-9826-456da5543121" containerID="8a7284c711847e031f0228f91927dbad499fbd3a0488142bd1e006a5564bbb6f" exitCode=0
Jul 13 15:30:13.509102 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.509055    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"
Jul 13 15:30:13.509102 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.509090    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm" event={"ID":"418e94c5-e3fd-4e6e-9826-456da5543121","Type":"ContainerDied","Data":"8a7284c711847e031f0228f91927dbad499fbd3a0488142bd1e006a5564bbb6f"}
Jul 13 15:30:13.509362 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.509141    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm" event={"ID":"418e94c5-e3fd-4e6e-9826-456da5543121","Type":"ContainerDied","Data":"92932019ea4beac13cf24b55efb308cc47c2034a591c427d6388b7db133e2e1d"}
Jul 13 15:30:13.509362 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.509164    2564 scope.go:117] "RemoveContainer" containerID="8a7284c711847e031f0228f91927dbad499fbd3a0488142bd1e006a5564bbb6f"
Jul 13 15:30:13.518122 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.518103    2564 scope.go:117] "RemoveContainer" containerID="8a7284c711847e031f0228f91927dbad499fbd3a0488142bd1e006a5564bbb6f"
Jul 13 15:30:13.518411 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:30:13.518385    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a7284c711847e031f0228f91927dbad499fbd3a0488142bd1e006a5564bbb6f\": container with ID starting with 8a7284c711847e031f0228f91927dbad499fbd3a0488142bd1e006a5564bbb6f not found: ID does not exist" containerID="8a7284c711847e031f0228f91927dbad499fbd3a0488142bd1e006a5564bbb6f"
Jul 13 15:30:13.518474 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.518420    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a7284c711847e031f0228f91927dbad499fbd3a0488142bd1e006a5564bbb6f"} err="failed to get container status \"8a7284c711847e031f0228f91927dbad499fbd3a0488142bd1e006a5564bbb6f\": rpc error: code = NotFound desc = could not find container \"8a7284c711847e031f0228f91927dbad499fbd3a0488142bd1e006a5564bbb6f\": container with ID starting with 8a7284c711847e031f0228f91927dbad499fbd3a0488142bd1e006a5564bbb6f not found: ID does not exist"
Jul 13 15:30:13.531035 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.531011    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"]
Jul 13 15:30:13.535233 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:13.535210    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-15d63b-755f8b9b76-52sxm"]
Jul 13 15:30:14.317402 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:14.317368    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="418e94c5-e3fd-4e6e-9826-456da5543121" path="/var/lib/kubelet/pods/418e94c5-e3fd-4e6e-9826-456da5543121/volumes"
Jul 13 15:30:15.696434 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:15.696375    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4" podUID="061ff01e-8445-458c-ba86-5e5a134259c6" containerName="main" containerID="cri-o://57054b7e01bead983d28d96b0e13ad7b1a53122a1cc72671780a8cee8e3dca30" gracePeriod=45
Jul 13 15:30:15.951909 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:15.951849    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:30:15.973049 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:15.973020    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/061ff01e-8445-458c-ba86-5e5a134259c6-tls-certs\") pod \"061ff01e-8445-458c-ba86-5e5a134259c6\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") "
Jul 13 15:30:15.973211 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:15.973095    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-tmp-dir\") pod \"061ff01e-8445-458c-ba86-5e5a134259c6\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") "
Jul 13 15:30:15.973211 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:15.973152    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-home\") pod \"061ff01e-8445-458c-ba86-5e5a134259c6\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") "
Jul 13 15:30:15.973211 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:15.973173    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kmwh\" (UniqueName: \"kubernetes.io/projected/061ff01e-8445-458c-ba86-5e5a134259c6-kube-api-access-5kmwh\") pod \"061ff01e-8445-458c-ba86-5e5a134259c6\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") "
Jul 13 15:30:15.973211 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:15.973206    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-dshm\") pod \"061ff01e-8445-458c-ba86-5e5a134259c6\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") "
Jul 13 15:30:15.973421 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:15.973327    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "061ff01e-8445-458c-ba86-5e5a134259c6" (UID: "061ff01e-8445-458c-ba86-5e5a134259c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:30:15.973474 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:15.973438    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-home" (OuterVolumeSpecName: "home") pod "061ff01e-8445-458c-ba86-5e5a134259c6" (UID: "061ff01e-8445-458c-ba86-5e5a134259c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:30:15.973535 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:15.973524    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:15.973586 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:15.973542    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:15.975720 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:15.975653    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061ff01e-8445-458c-ba86-5e5a134259c6-kube-api-access-5kmwh" (OuterVolumeSpecName: "kube-api-access-5kmwh") pod "061ff01e-8445-458c-ba86-5e5a134259c6" (UID: "061ff01e-8445-458c-ba86-5e5a134259c6"). InnerVolumeSpecName "kube-api-access-5kmwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:30:15.975828 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:15.975724    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-dshm" (OuterVolumeSpecName: "dshm") pod "061ff01e-8445-458c-ba86-5e5a134259c6" (UID: "061ff01e-8445-458c-ba86-5e5a134259c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:30:15.975828 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:15.975811    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061ff01e-8445-458c-ba86-5e5a134259c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "061ff01e-8445-458c-ba86-5e5a134259c6" (UID: "061ff01e-8445-458c-ba86-5e5a134259c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:30:16.074286 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.074249    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-model-cache\") pod \"061ff01e-8445-458c-ba86-5e5a134259c6\" (UID: \"061ff01e-8445-458c-ba86-5e5a134259c6\") "
Jul 13 15:30:16.074461 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.074446    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kmwh\" (UniqueName: \"kubernetes.io/projected/061ff01e-8445-458c-ba86-5e5a134259c6-kube-api-access-5kmwh\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:16.074461 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.074459    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:16.074545 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.074469    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/061ff01e-8445-458c-ba86-5e5a134259c6-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:16.074545 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.074513    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "061ff01e-8445-458c-ba86-5e5a134259c6" (UID: "061ff01e-8445-458c-ba86-5e5a134259c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:30:16.175038 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.175005    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/061ff01e-8445-458c-ba86-5e5a134259c6-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:16.525169 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.525133    2564 generic.go:356] "Generic (PLEG): container finished" podID="061ff01e-8445-458c-ba86-5e5a134259c6" containerID="57054b7e01bead983d28d96b0e13ad7b1a53122a1cc72671780a8cee8e3dca30" exitCode=0
Jul 13 15:30:16.525353 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.525179    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4" event={"ID":"061ff01e-8445-458c-ba86-5e5a134259c6","Type":"ContainerDied","Data":"57054b7e01bead983d28d96b0e13ad7b1a53122a1cc72671780a8cee8e3dca30"}
Jul 13 15:30:16.525353 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.525188    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"
Jul 13 15:30:16.525353 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.525219    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4" event={"ID":"061ff01e-8445-458c-ba86-5e5a134259c6","Type":"ContainerDied","Data":"c011dd1f97ebd29d3e4874160705566e0322e20713dbd0e6168986793587d513"}
Jul 13 15:30:16.525353 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.525238    2564 scope.go:117] "RemoveContainer" containerID="57054b7e01bead983d28d96b0e13ad7b1a53122a1cc72671780a8cee8e3dca30"
Jul 13 15:30:16.534200 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.534180    2564 scope.go:117] "RemoveContainer" containerID="57054b7e01bead983d28d96b0e13ad7b1a53122a1cc72671780a8cee8e3dca30"
Jul 13 15:30:16.534458 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:30:16.534438    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57054b7e01bead983d28d96b0e13ad7b1a53122a1cc72671780a8cee8e3dca30\": container with ID starting with 57054b7e01bead983d28d96b0e13ad7b1a53122a1cc72671780a8cee8e3dca30 not found: ID does not exist" containerID="57054b7e01bead983d28d96b0e13ad7b1a53122a1cc72671780a8cee8e3dca30"
Jul 13 15:30:16.534565 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.534467    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57054b7e01bead983d28d96b0e13ad7b1a53122a1cc72671780a8cee8e3dca30"} err="failed to get container status \"57054b7e01bead983d28d96b0e13ad7b1a53122a1cc72671780a8cee8e3dca30\": rpc error: code = NotFound desc = could not find container \"57054b7e01bead983d28d96b0e13ad7b1a53122a1cc72671780a8cee8e3dca30\": container with ID starting with 57054b7e01bead983d28d96b0e13ad7b1a53122a1cc72671780a8cee8e3dca30 not found: ID does not exist"
Jul 13 15:30:16.542038 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.542014    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"]
Jul 13 15:30:16.546298 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:16.546273    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-15d63b/test-model-15d63b-kserve-59d9685d67-z4rj4"]
Jul 13 15:30:17.042817 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.042777    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6" podUID="c83eec51-9c3a-4355-a3e0-aa34087a8fbb" containerName="main" containerID="cri-o://c855c125cb12ce18a2198efab91daddda95627fbeda44fe52c43301ff538054f" gracePeriod=45
Jul 13 15:30:17.296406 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.296349    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:30:17.384423 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.384388    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-tls-certs\") pod \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") "
Jul 13 15:30:17.384423 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.384431    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkhbj\" (UniqueName: \"kubernetes.io/projected/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-kube-api-access-mkhbj\") pod \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") "
Jul 13 15:30:17.384720 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.384496    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-dshm\") pod \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") "
Jul 13 15:30:17.384720 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.384516    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-home\") pod \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") "
Jul 13 15:30:17.384720 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.384571    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-model-cache\") pod \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") "
Jul 13 15:30:17.384720 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.384611    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-tmp-dir\") pod \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\" (UID: \"c83eec51-9c3a-4355-a3e0-aa34087a8fbb\") "
Jul 13 15:30:17.384926 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.384889    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-model-cache" (OuterVolumeSpecName: "model-cache") pod "c83eec51-9c3a-4355-a3e0-aa34087a8fbb" (UID: "c83eec51-9c3a-4355-a3e0-aa34087a8fbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:30:17.385050 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.385025    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-home" (OuterVolumeSpecName: "home") pod "c83eec51-9c3a-4355-a3e0-aa34087a8fbb" (UID: "c83eec51-9c3a-4355-a3e0-aa34087a8fbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:30:17.385127 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.385031    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c83eec51-9c3a-4355-a3e0-aa34087a8fbb" (UID: "c83eec51-9c3a-4355-a3e0-aa34087a8fbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:30:17.386961 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.386923    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-kube-api-access-mkhbj" (OuterVolumeSpecName: "kube-api-access-mkhbj") pod "c83eec51-9c3a-4355-a3e0-aa34087a8fbb" (UID: "c83eec51-9c3a-4355-a3e0-aa34087a8fbb"). InnerVolumeSpecName "kube-api-access-mkhbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:30:17.387068 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.386988    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-dshm" (OuterVolumeSpecName: "dshm") pod "c83eec51-9c3a-4355-a3e0-aa34087a8fbb" (UID: "c83eec51-9c3a-4355-a3e0-aa34087a8fbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:30:17.387068 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.387008    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c83eec51-9c3a-4355-a3e0-aa34087a8fbb" (UID: "c83eec51-9c3a-4355-a3e0-aa34087a8fbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:30:17.485482 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.485443    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:17.485482 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.485472    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:17.485482 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.485482    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:17.485482 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.485492    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:17.485786 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.485499    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:17.485786 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.485508    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkhbj\" (UniqueName: \"kubernetes.io/projected/c83eec51-9c3a-4355-a3e0-aa34087a8fbb-kube-api-access-mkhbj\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:17.535214 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.532070    2564 generic.go:356] "Generic (PLEG): container finished" podID="c83eec51-9c3a-4355-a3e0-aa34087a8fbb" containerID="c855c125cb12ce18a2198efab91daddda95627fbeda44fe52c43301ff538054f" exitCode=0
Jul 13 15:30:17.535214 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.532186    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6" event={"ID":"c83eec51-9c3a-4355-a3e0-aa34087a8fbb","Type":"ContainerDied","Data":"c855c125cb12ce18a2198efab91daddda95627fbeda44fe52c43301ff538054f"}
Jul 13 15:30:17.535214 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.532223    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6" event={"ID":"c83eec51-9c3a-4355-a3e0-aa34087a8fbb","Type":"ContainerDied","Data":"48ac5c3673806a40b7e733e1d567955d0761497d3897c755a41a9237003ea138"}
Jul 13 15:30:17.535214 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.532248    2564 scope.go:117] "RemoveContainer" containerID="c855c125cb12ce18a2198efab91daddda95627fbeda44fe52c43301ff538054f"
Jul 13 15:30:17.535214 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.532358    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"
Jul 13 15:30:17.545365 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.545345    2564 scope.go:117] "RemoveContainer" containerID="c855c125cb12ce18a2198efab91daddda95627fbeda44fe52c43301ff538054f"
Jul 13 15:30:17.545626 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:30:17.545609    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c855c125cb12ce18a2198efab91daddda95627fbeda44fe52c43301ff538054f\": container with ID starting with c855c125cb12ce18a2198efab91daddda95627fbeda44fe52c43301ff538054f not found: ID does not exist" containerID="c855c125cb12ce18a2198efab91daddda95627fbeda44fe52c43301ff538054f"
Jul 13 15:30:17.545725 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.545640    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c855c125cb12ce18a2198efab91daddda95627fbeda44fe52c43301ff538054f"} err="failed to get container status \"c855c125cb12ce18a2198efab91daddda95627fbeda44fe52c43301ff538054f\": rpc error: code = NotFound desc = could not find container \"c855c125cb12ce18a2198efab91daddda95627fbeda44fe52c43301ff538054f\": container with ID starting with c855c125cb12ce18a2198efab91daddda95627fbeda44fe52c43301ff538054f not found: ID does not exist"
Jul 13 15:30:17.555833 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.555770    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"]
Jul 13 15:30:17.559086 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:17.559065    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f68651/test-model-f68651-kserve-6c7577f89-9pqw6"]
Jul 13 15:30:18.317096 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:18.317065    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="061ff01e-8445-458c-ba86-5e5a134259c6" path="/var/lib/kubelet/pods/061ff01e-8445-458c-ba86-5e5a134259c6/volumes"
Jul 13 15:30:18.317466 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:18.317449    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c83eec51-9c3a-4355-a3e0-aa34087a8fbb" path="/var/lib/kubelet/pods/c83eec51-9c3a-4355-a3e0-aa34087a8fbb/volumes"
Jul 13 15:30:38.690350 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:38.690320    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"]
Jul 13 15:30:38.690824 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:38.690579    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j" podUID="92c32263-2f6b-4ec7-b070-6dd409049589" containerName="maas-api" containerID="cri-o://025dc6c218565a3e1d1e0aa50654361b27076d76c5cfa0ccc50bf2f6b432bdf3" gracePeriod=30
Jul 13 15:30:38.933109 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:38.933082    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:30:39.083757 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.083627    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c32263-2f6b-4ec7-b070-6dd409049589-maas-api-tls\") pod \"92c32263-2f6b-4ec7-b070-6dd409049589\" (UID: \"92c32263-2f6b-4ec7-b070-6dd409049589\") "
Jul 13 15:30:39.083757 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.083687    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c32263-2f6b-4ec7-b070-6dd409049589-openshift-service-ca\") pod \"92c32263-2f6b-4ec7-b070-6dd409049589\" (UID: \"92c32263-2f6b-4ec7-b070-6dd409049589\") "
Jul 13 15:30:39.083757 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.083761    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7nk4\" (UniqueName: \"kubernetes.io/projected/92c32263-2f6b-4ec7-b070-6dd409049589-kube-api-access-h7nk4\") pod \"92c32263-2f6b-4ec7-b070-6dd409049589\" (UID: \"92c32263-2f6b-4ec7-b070-6dd409049589\") "
Jul 13 15:30:39.084122 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.084092    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c32263-2f6b-4ec7-b070-6dd409049589-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "92c32263-2f6b-4ec7-b070-6dd409049589" (UID: "92c32263-2f6b-4ec7-b070-6dd409049589"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:30:39.085937 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.085911    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92c32263-2f6b-4ec7-b070-6dd409049589-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "92c32263-2f6b-4ec7-b070-6dd409049589" (UID: "92c32263-2f6b-4ec7-b070-6dd409049589"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:30:39.086078 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.086054    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c32263-2f6b-4ec7-b070-6dd409049589-kube-api-access-h7nk4" (OuterVolumeSpecName: "kube-api-access-h7nk4") pod "92c32263-2f6b-4ec7-b070-6dd409049589" (UID: "92c32263-2f6b-4ec7-b070-6dd409049589"). InnerVolumeSpecName "kube-api-access-h7nk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:30:39.184809 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.184763    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c32263-2f6b-4ec7-b070-6dd409049589-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:39.184809 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.184803    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c32263-2f6b-4ec7-b070-6dd409049589-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:39.184809 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.184814    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7nk4\" (UniqueName: \"kubernetes.io/projected/92c32263-2f6b-4ec7-b070-6dd409049589-kube-api-access-h7nk4\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:30:39.641856 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.641816    2564 generic.go:356] "Generic (PLEG): container finished" podID="92c32263-2f6b-4ec7-b070-6dd409049589" containerID="025dc6c218565a3e1d1e0aa50654361b27076d76c5cfa0ccc50bf2f6b432bdf3" exitCode=0
Jul 13 15:30:39.642065 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.641861    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"
Jul 13 15:30:39.642065 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.641894    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j" event={"ID":"92c32263-2f6b-4ec7-b070-6dd409049589","Type":"ContainerDied","Data":"025dc6c218565a3e1d1e0aa50654361b27076d76c5cfa0ccc50bf2f6b432bdf3"}
Jul 13 15:30:39.642065 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.641946    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j" event={"ID":"92c32263-2f6b-4ec7-b070-6dd409049589","Type":"ContainerDied","Data":"76af43a61688ad4a8a8c46e4c9f20f41916f5ba0b6a34d9c3c408fdfb5173e8a"}
Jul 13 15:30:39.642065 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.641974    2564 scope.go:117] "RemoveContainer" containerID="025dc6c218565a3e1d1e0aa50654361b27076d76c5cfa0ccc50bf2f6b432bdf3"
Jul 13 15:30:39.651116 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.651098    2564 scope.go:117] "RemoveContainer" containerID="025dc6c218565a3e1d1e0aa50654361b27076d76c5cfa0ccc50bf2f6b432bdf3"
Jul 13 15:30:39.651397 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:30:39.651376    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025dc6c218565a3e1d1e0aa50654361b27076d76c5cfa0ccc50bf2f6b432bdf3\": container with ID starting with 025dc6c218565a3e1d1e0aa50654361b27076d76c5cfa0ccc50bf2f6b432bdf3 not found: ID does not exist" containerID="025dc6c218565a3e1d1e0aa50654361b27076d76c5cfa0ccc50bf2f6b432bdf3"
Jul 13 15:30:39.651459 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.651409    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025dc6c218565a3e1d1e0aa50654361b27076d76c5cfa0ccc50bf2f6b432bdf3"} err="failed to get container status \"025dc6c218565a3e1d1e0aa50654361b27076d76c5cfa0ccc50bf2f6b432bdf3\": rpc error: code = NotFound desc = could not find container \"025dc6c218565a3e1d1e0aa50654361b27076d76c5cfa0ccc50bf2f6b432bdf3\": container with ID starting with 025dc6c218565a3e1d1e0aa50654361b27076d76c5cfa0ccc50bf2f6b432bdf3 not found: ID does not exist"
Jul 13 15:30:39.663260 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.663236    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"]
Jul 13 15:30:39.666574 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:39.666555    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f68651-744d645f7d-vzv7j"]
Jul 13 15:30:40.316740 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:40.316689    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92c32263-2f6b-4ec7-b070-6dd409049589" path="/var/lib/kubelet/pods/92c32263-2f6b-4ec7-b070-6dd409049589/volumes"
Jul 13 15:30:58.373790 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.373756    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"]
Jul 13 15:30:58.374306 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.374130    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c83eec51-9c3a-4355-a3e0-aa34087a8fbb" containerName="main"
Jul 13 15:30:58.374306 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.374142    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c83eec51-9c3a-4355-a3e0-aa34087a8fbb" containerName="main"
Jul 13 15:30:58.374306 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.374152    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c32263-2f6b-4ec7-b070-6dd409049589" containerName="maas-api"
Jul 13 15:30:58.374306 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.374157    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c32263-2f6b-4ec7-b070-6dd409049589" containerName="maas-api"
Jul 13 15:30:58.374306 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.374172    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="418e94c5-e3fd-4e6e-9826-456da5543121" containerName="maas-api"
Jul 13 15:30:58.374306 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.374178    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="418e94c5-e3fd-4e6e-9826-456da5543121" containerName="maas-api"
Jul 13 15:30:58.374306 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.374190    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061ff01e-8445-458c-ba86-5e5a134259c6" containerName="main"
Jul 13 15:30:58.374306 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.374194    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="061ff01e-8445-458c-ba86-5e5a134259c6" containerName="main"
Jul 13 15:30:58.374306 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.374253    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c83eec51-9c3a-4355-a3e0-aa34087a8fbb" containerName="main"
Jul 13 15:30:58.374306 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.374261    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="061ff01e-8445-458c-ba86-5e5a134259c6" containerName="main"
Jul 13 15:30:58.374306 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.374267    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="418e94c5-e3fd-4e6e-9826-456da5543121" containerName="maas-api"
Jul 13 15:30:58.374306 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.374273    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c32263-2f6b-4ec7-b070-6dd409049589" containerName="maas-api"
Jul 13 15:30:58.378028 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.378007    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:30:58.381551 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.381530    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvc6k\""
Jul 13 15:30:58.381924 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.381905    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 15:30:58.384304 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.384278    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4b866e-serving-cert\""
Jul 13 15:30:58.387458 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.387436    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 15:30:58.414310 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.414274    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"]
Jul 13 15:30:58.459230 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.459195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04e6fde9-4107-460e-b3b7-d0da663fb95f-maas-api-tls\") pod \"maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk\" (UID: \"04e6fde9-4107-460e-b3b7-d0da663fb95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:30:58.459407 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.459257    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94qkg\" (UniqueName: \"kubernetes.io/projected/04e6fde9-4107-460e-b3b7-d0da663fb95f-kube-api-access-94qkg\") pod \"maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk\" (UID: \"04e6fde9-4107-460e-b3b7-d0da663fb95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:30:58.459407 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.459284    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e6fde9-4107-460e-b3b7-d0da663fb95f-openshift-service-ca\") pod \"maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk\" (UID: \"04e6fde9-4107-460e-b3b7-d0da663fb95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:30:58.560168 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.560135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qkg\" (UniqueName: \"kubernetes.io/projected/04e6fde9-4107-460e-b3b7-d0da663fb95f-kube-api-access-94qkg\") pod \"maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk\" (UID: \"04e6fde9-4107-460e-b3b7-d0da663fb95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:30:58.560373 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.560184    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e6fde9-4107-460e-b3b7-d0da663fb95f-openshift-service-ca\") pod \"maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk\" (UID: \"04e6fde9-4107-460e-b3b7-d0da663fb95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:30:58.560373 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.560240    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04e6fde9-4107-460e-b3b7-d0da663fb95f-maas-api-tls\") pod \"maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk\" (UID: \"04e6fde9-4107-460e-b3b7-d0da663fb95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:30:58.560373 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:30:58.560343    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-serving-cert: secret "maas-api-e2e-api-a-4b866e-serving-cert" not found
Jul 13 15:30:58.560538 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:30:58.560400    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e6fde9-4107-460e-b3b7-d0da663fb95f-maas-api-tls podName:04e6fde9-4107-460e-b3b7-d0da663fb95f nodeName:}" failed. No retries permitted until 2026-07-13 15:30:59.060383154 +0000 UTC m=+2285.257536079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/04e6fde9-4107-460e-b3b7-d0da663fb95f-maas-api-tls") pod "maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk" (UID: "04e6fde9-4107-460e-b3b7-d0da663fb95f") : secret "maas-api-e2e-api-a-4b866e-serving-cert" not found
Jul 13 15:30:58.560877 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.560859    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e6fde9-4107-460e-b3b7-d0da663fb95f-openshift-service-ca\") pod \"maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk\" (UID: \"04e6fde9-4107-460e-b3b7-d0da663fb95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:30:58.573244 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:58.573220    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94qkg\" (UniqueName: \"kubernetes.io/projected/04e6fde9-4107-460e-b3b7-d0da663fb95f-kube-api-access-94qkg\") pod \"maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk\" (UID: \"04e6fde9-4107-460e-b3b7-d0da663fb95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:30:59.063570 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:59.063536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04e6fde9-4107-460e-b3b7-d0da663fb95f-maas-api-tls\") pod \"maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk\" (UID: \"04e6fde9-4107-460e-b3b7-d0da663fb95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:30:59.066159 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:59.066130    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04e6fde9-4107-460e-b3b7-d0da663fb95f-maas-api-tls\") pod \"maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk\" (UID: \"04e6fde9-4107-460e-b3b7-d0da663fb95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:30:59.288905 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:59.288869    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:30:59.425403 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:59.425245    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"]
Jul 13 15:30:59.427979 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:30:59.427950    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e6fde9_4107_460e_b3b7_d0da663fb95f.slice/crio-e6a34aeba9caa1c1c9c42f542ca3113dd5160e3e8b7a9a7bac77e8ef92a8bc58 WatchSource:0}: Error finding container e6a34aeba9caa1c1c9c42f542ca3113dd5160e3e8b7a9a7bac77e8ef92a8bc58: Status 404 returned error can't find the container with id e6a34aeba9caa1c1c9c42f542ca3113dd5160e3e8b7a9a7bac77e8ef92a8bc58
Jul 13 15:30:59.429421 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:59.429402    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 15:30:59.743888 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:30:59.743851    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk" event={"ID":"04e6fde9-4107-460e-b3b7-d0da663fb95f","Type":"ContainerStarted","Data":"e6a34aeba9caa1c1c9c42f542ca3113dd5160e3e8b7a9a7bac77e8ef92a8bc58"}
Jul 13 15:31:00.749950 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:00.749916    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk" event={"ID":"04e6fde9-4107-460e-b3b7-d0da663fb95f","Type":"ContainerStarted","Data":"bc32bd47832533d7cb31bf430d0a15c2d92d72e1aedfa5a769a91264ee84bb6b"}
Jul 13 15:31:00.750346 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:00.750043    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:31:00.769884 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:00.769837    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk" podStartSLOduration=2.202335094 podStartE2EDuration="2.769822795s" totalImagesPullingTime="567.487701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:31:00.76810982 +0000 UTC m=+2286.965262770" watchObservedRunningTime="2026-07-13 15:31:00.769822795 +0000 UTC m=+2286.966975743"
Jul 13 15:31:06.759509 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:06.759479    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:31:06.907657 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:06.907615    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"]
Jul 13 15:31:06.919542 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:06.919513    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:06.922951 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:06.922928    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d3da1a-serving-cert\""
Jul 13 15:31:06.926299 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:06.926275    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"]
Jul 13 15:31:07.032815 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:07.032735    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz\" (UID: \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:07.032815 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:07.032792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-maas-api-tls\") pod \"maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz\" (UID: \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:07.032993 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:07.032828    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87f5t\" (UniqueName: \"kubernetes.io/projected/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-kube-api-access-87f5t\") pod \"maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz\" (UID: \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:07.134138 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:07.134105    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz\" (UID: \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:07.134295 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:07.134165    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-maas-api-tls\") pod \"maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz\" (UID: \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:07.134295 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:07.134197    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87f5t\" (UniqueName: \"kubernetes.io/projected/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-kube-api-access-87f5t\") pod \"maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz\" (UID: \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:07.134366 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:31:07.134336    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-serving-cert: secret "maas-api-e2e-api-b-d3da1a-serving-cert" not found
Jul 13 15:31:07.134415 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:31:07.134404    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-maas-api-tls podName:298feea9-0cc8-4a12-93ce-3b86e77ffc7e nodeName:}" failed. No retries permitted until 2026-07-13 15:31:07.634384522 +0000 UTC m=+2293.831537448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-maas-api-tls") pod "maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz" (UID: "298feea9-0cc8-4a12-93ce-3b86e77ffc7e") : secret "maas-api-e2e-api-b-d3da1a-serving-cert" not found
Jul 13 15:31:07.134750 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:07.134730    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz\" (UID: \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:07.152870 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:07.152848    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87f5t\" (UniqueName: \"kubernetes.io/projected/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-kube-api-access-87f5t\") pod \"maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz\" (UID: \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:07.639623 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:07.639579    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-maas-api-tls\") pod \"maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz\" (UID: \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:07.642391 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:07.642367    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-maas-api-tls\") pod \"maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz\" (UID: \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:07.832003 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:07.831962    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:07.972004 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:07.971972    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"]
Jul 13 15:31:07.975473 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:31:07.975438    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298feea9_0cc8_4a12_93ce_3b86e77ffc7e.slice/crio-df4987038761728ef6f1ea6ac09d269f3354fb73575fde4990ecbc25961f2ff8 WatchSource:0}: Error finding container df4987038761728ef6f1ea6ac09d269f3354fb73575fde4990ecbc25961f2ff8: Status 404 returned error can't find the container with id df4987038761728ef6f1ea6ac09d269f3354fb73575fde4990ecbc25961f2ff8
Jul 13 15:31:08.790818 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:08.790724    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz" event={"ID":"298feea9-0cc8-4a12-93ce-3b86e77ffc7e","Type":"ContainerStarted","Data":"dfd08ec0f19b93bf702c1f20440f6b63d497aff5ed88458cfe24468a0b8c9517"}
Jul 13 15:31:08.790818 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:08.790779    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz" event={"ID":"298feea9-0cc8-4a12-93ce-3b86e77ffc7e","Type":"ContainerStarted","Data":"df4987038761728ef6f1ea6ac09d269f3354fb73575fde4990ecbc25961f2ff8"}
Jul 13 15:31:08.791030 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:08.790827    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:08.807755 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:08.807689    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz" podStartSLOduration=2.382734218 podStartE2EDuration="2.807674242s" totalImagesPullingTime="424.940024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:31:08.806207742 +0000 UTC m=+2295.003360689" watchObservedRunningTime="2026-07-13 15:31:08.807674242 +0000 UTC m=+2295.004827213"
Jul 13 15:31:14.800591 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:14.800560    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:29.610019 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.609982    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"]
Jul 13 15:31:29.610530 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.610253    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk" podUID="04e6fde9-4107-460e-b3b7-d0da663fb95f" containerName="maas-api" containerID="cri-o://bc32bd47832533d7cb31bf430d0a15c2d92d72e1aedfa5a769a91264ee84bb6b" gracePeriod=30
Jul 13 15:31:29.857588 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.857563    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:31:29.897091 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.897061    2564 generic.go:356] "Generic (PLEG): container finished" podID="04e6fde9-4107-460e-b3b7-d0da663fb95f" containerID="bc32bd47832533d7cb31bf430d0a15c2d92d72e1aedfa5a769a91264ee84bb6b" exitCode=0
Jul 13 15:31:29.897298 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.897104    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"
Jul 13 15:31:29.897298 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.897160    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk" event={"ID":"04e6fde9-4107-460e-b3b7-d0da663fb95f","Type":"ContainerDied","Data":"bc32bd47832533d7cb31bf430d0a15c2d92d72e1aedfa5a769a91264ee84bb6b"}
Jul 13 15:31:29.897298 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.897232    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk" event={"ID":"04e6fde9-4107-460e-b3b7-d0da663fb95f","Type":"ContainerDied","Data":"e6a34aeba9caa1c1c9c42f542ca3113dd5160e3e8b7a9a7bac77e8ef92a8bc58"}
Jul 13 15:31:29.897298 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.897262    2564 scope.go:117] "RemoveContainer" containerID="bc32bd47832533d7cb31bf430d0a15c2d92d72e1aedfa5a769a91264ee84bb6b"
Jul 13 15:31:29.905856 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.905840    2564 scope.go:117] "RemoveContainer" containerID="bc32bd47832533d7cb31bf430d0a15c2d92d72e1aedfa5a769a91264ee84bb6b"
Jul 13 15:31:29.906139 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:31:29.906115    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc32bd47832533d7cb31bf430d0a15c2d92d72e1aedfa5a769a91264ee84bb6b\": container with ID starting with bc32bd47832533d7cb31bf430d0a15c2d92d72e1aedfa5a769a91264ee84bb6b not found: ID does not exist" containerID="bc32bd47832533d7cb31bf430d0a15c2d92d72e1aedfa5a769a91264ee84bb6b"
Jul 13 15:31:29.906209 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.906149    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc32bd47832533d7cb31bf430d0a15c2d92d72e1aedfa5a769a91264ee84bb6b"} err="failed to get container status \"bc32bd47832533d7cb31bf430d0a15c2d92d72e1aedfa5a769a91264ee84bb6b\": rpc error: code = NotFound desc = could not find container \"bc32bd47832533d7cb31bf430d0a15c2d92d72e1aedfa5a769a91264ee84bb6b\": container with ID starting with bc32bd47832533d7cb31bf430d0a15c2d92d72e1aedfa5a769a91264ee84bb6b not found: ID does not exist"
Jul 13 15:31:29.935604 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.935582    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e6fde9-4107-460e-b3b7-d0da663fb95f-openshift-service-ca\") pod \"04e6fde9-4107-460e-b3b7-d0da663fb95f\" (UID: \"04e6fde9-4107-460e-b3b7-d0da663fb95f\") "
Jul 13 15:31:29.935748 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.935670    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04e6fde9-4107-460e-b3b7-d0da663fb95f-maas-api-tls\") pod \"04e6fde9-4107-460e-b3b7-d0da663fb95f\" (UID: \"04e6fde9-4107-460e-b3b7-d0da663fb95f\") "
Jul 13 15:31:29.935748 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.935707    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94qkg\" (UniqueName: \"kubernetes.io/projected/04e6fde9-4107-460e-b3b7-d0da663fb95f-kube-api-access-94qkg\") pod \"04e6fde9-4107-460e-b3b7-d0da663fb95f\" (UID: \"04e6fde9-4107-460e-b3b7-d0da663fb95f\") "
Jul 13 15:31:29.936004 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.935976    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e6fde9-4107-460e-b3b7-d0da663fb95f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "04e6fde9-4107-460e-b3b7-d0da663fb95f" (UID: "04e6fde9-4107-460e-b3b7-d0da663fb95f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:31:29.937879 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.937856    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e6fde9-4107-460e-b3b7-d0da663fb95f-kube-api-access-94qkg" (OuterVolumeSpecName: "kube-api-access-94qkg") pod "04e6fde9-4107-460e-b3b7-d0da663fb95f" (UID: "04e6fde9-4107-460e-b3b7-d0da663fb95f"). InnerVolumeSpecName "kube-api-access-94qkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:31:29.937879 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:29.937872    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e6fde9-4107-460e-b3b7-d0da663fb95f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "04e6fde9-4107-460e-b3b7-d0da663fb95f" (UID: "04e6fde9-4107-460e-b3b7-d0da663fb95f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:31:30.036451 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:30.036414    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e6fde9-4107-460e-b3b7-d0da663fb95f-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:31:30.036451 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:30.036445    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04e6fde9-4107-460e-b3b7-d0da663fb95f-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:31:30.036451 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:30.036454    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94qkg\" (UniqueName: \"kubernetes.io/projected/04e6fde9-4107-460e-b3b7-d0da663fb95f-kube-api-access-94qkg\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:31:30.218756 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:30.218724    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"]
Jul 13 15:31:30.222886 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:30.222860    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b866e-7f48b5dfb9-mf7tk"]
Jul 13 15:31:30.319344 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:30.319294    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04e6fde9-4107-460e-b3b7-d0da663fb95f" path="/var/lib/kubelet/pods/04e6fde9-4107-460e-b3b7-d0da663fb95f/volumes"
Jul 13 15:31:55.991248 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:55.991213    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"]
Jul 13 15:31:55.991765 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:55.991464    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz" podUID="298feea9-0cc8-4a12-93ce-3b86e77ffc7e" containerName="maas-api" containerID="cri-o://dfd08ec0f19b93bf702c1f20440f6b63d497aff5ed88458cfe24468a0b8c9517" gracePeriod=30
Jul 13 15:31:56.239128 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:56.239102    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:56.274114 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:56.274039    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-openshift-service-ca\") pod \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\" (UID: \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\") "
Jul 13 15:31:56.274114 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:56.274088    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-maas-api-tls\") pod \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\" (UID: \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\") "
Jul 13 15:31:56.274291 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:56.274175    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87f5t\" (UniqueName: \"kubernetes.io/projected/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-kube-api-access-87f5t\") pod \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\" (UID: \"298feea9-0cc8-4a12-93ce-3b86e77ffc7e\") "
Jul 13 15:31:56.274431 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:56.274407    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "298feea9-0cc8-4a12-93ce-3b86e77ffc7e" (UID: "298feea9-0cc8-4a12-93ce-3b86e77ffc7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:31:56.276337 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:56.276306    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "298feea9-0cc8-4a12-93ce-3b86e77ffc7e" (UID: "298feea9-0cc8-4a12-93ce-3b86e77ffc7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:31:56.276449 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:56.276412    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-kube-api-access-87f5t" (OuterVolumeSpecName: "kube-api-access-87f5t") pod "298feea9-0cc8-4a12-93ce-3b86e77ffc7e" (UID: "298feea9-0cc8-4a12-93ce-3b86e77ffc7e"). InnerVolumeSpecName "kube-api-access-87f5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:31:56.375153 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:56.375122    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:31:56.375153 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:56.375146    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87f5t\" (UniqueName: \"kubernetes.io/projected/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-kube-api-access-87f5t\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:31:56.375153 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:56.375156    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/298feea9-0cc8-4a12-93ce-3b86e77ffc7e-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:31:57.027442 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:57.027407    2564 generic.go:356] "Generic (PLEG): container finished" podID="298feea9-0cc8-4a12-93ce-3b86e77ffc7e" containerID="dfd08ec0f19b93bf702c1f20440f6b63d497aff5ed88458cfe24468a0b8c9517" exitCode=0
Jul 13 15:31:57.027943 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:57.027469    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"
Jul 13 15:31:57.027943 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:57.027486    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz" event={"ID":"298feea9-0cc8-4a12-93ce-3b86e77ffc7e","Type":"ContainerDied","Data":"dfd08ec0f19b93bf702c1f20440f6b63d497aff5ed88458cfe24468a0b8c9517"}
Jul 13 15:31:57.027943 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:57.027542    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz" event={"ID":"298feea9-0cc8-4a12-93ce-3b86e77ffc7e","Type":"ContainerDied","Data":"df4987038761728ef6f1ea6ac09d269f3354fb73575fde4990ecbc25961f2ff8"}
Jul 13 15:31:57.027943 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:57.027576    2564 scope.go:117] "RemoveContainer" containerID="dfd08ec0f19b93bf702c1f20440f6b63d497aff5ed88458cfe24468a0b8c9517"
Jul 13 15:31:57.036349 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:57.036329    2564 scope.go:117] "RemoveContainer" containerID="dfd08ec0f19b93bf702c1f20440f6b63d497aff5ed88458cfe24468a0b8c9517"
Jul 13 15:31:57.036623 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:31:57.036595    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd08ec0f19b93bf702c1f20440f6b63d497aff5ed88458cfe24468a0b8c9517\": container with ID starting with dfd08ec0f19b93bf702c1f20440f6b63d497aff5ed88458cfe24468a0b8c9517 not found: ID does not exist" containerID="dfd08ec0f19b93bf702c1f20440f6b63d497aff5ed88458cfe24468a0b8c9517"
Jul 13 15:31:57.036742 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:57.036628    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd08ec0f19b93bf702c1f20440f6b63d497aff5ed88458cfe24468a0b8c9517"} err="failed to get container status \"dfd08ec0f19b93bf702c1f20440f6b63d497aff5ed88458cfe24468a0b8c9517\": rpc error: code = NotFound desc = could not find container \"dfd08ec0f19b93bf702c1f20440f6b63d497aff5ed88458cfe24468a0b8c9517\": container with ID starting with dfd08ec0f19b93bf702c1f20440f6b63d497aff5ed88458cfe24468a0b8c9517 not found: ID does not exist"
Jul 13 15:31:57.044050 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:57.044026    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"]
Jul 13 15:31:57.048043 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:57.048023    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3da1a-667997bd99-5dtqz"]
Jul 13 15:31:58.316422 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:31:58.316386    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="298feea9-0cc8-4a12-93ce-3b86e77ffc7e" path="/var/lib/kubelet/pods/298feea9-0cc8-4a12-93ce-3b86e77ffc7e/volumes"
Jul 13 15:32:15.772624 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.772582    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"]
Jul 13 15:32:15.773159 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.773106    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="298feea9-0cc8-4a12-93ce-3b86e77ffc7e" containerName="maas-api"
Jul 13 15:32:15.773159 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.773127    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="298feea9-0cc8-4a12-93ce-3b86e77ffc7e" containerName="maas-api"
Jul 13 15:32:15.773159 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.773158    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e6fde9-4107-460e-b3b7-d0da663fb95f" containerName="maas-api"
Jul 13 15:32:15.773337 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.773167    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e6fde9-4107-460e-b3b7-d0da663fb95f" containerName="maas-api"
Jul 13 15:32:15.773337 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.773259    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="298feea9-0cc8-4a12-93ce-3b86e77ffc7e" containerName="maas-api"
Jul 13 15:32:15.773337 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.773286    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e6fde9-4107-460e-b3b7-d0da663fb95f" containerName="maas-api"
Jul 13 15:32:15.776510 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.776485    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:32:15.779139 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.779117    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 15:32:15.780037 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.780017    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-88b181-serving-cert\""
Jul 13 15:32:15.780109 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.780032    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvc6k\""
Jul 13 15:32:15.780109 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.780043    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 15:32:15.786913 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.786892    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"]
Jul 13 15:32:15.953399 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.953358    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqkj8\" (UniqueName: \"kubernetes.io/projected/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-kube-api-access-kqkj8\") pod \"maas-api-e2e-shared-a-88b181-965b9ff87-4xs24\" (UID: \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:32:15.953596 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.953413    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-maas-api-tls\") pod \"maas-api-e2e-shared-a-88b181-965b9ff87-4xs24\" (UID: \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:32:15.953596 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:15.953500    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-88b181-965b9ff87-4xs24\" (UID: \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:32:16.054838 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:16.054745    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-88b181-965b9ff87-4xs24\" (UID: \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:32:16.054838 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:16.054806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqkj8\" (UniqueName: \"kubernetes.io/projected/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-kube-api-access-kqkj8\") pod \"maas-api-e2e-shared-a-88b181-965b9ff87-4xs24\" (UID: \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:32:16.055059 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:16.054846    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-maas-api-tls\") pod \"maas-api-e2e-shared-a-88b181-965b9ff87-4xs24\" (UID: \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:32:16.055059 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:32:16.054985    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-serving-cert: secret "maas-api-e2e-shared-a-88b181-serving-cert" not found
Jul 13 15:32:16.055059 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:32:16.055060    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-maas-api-tls podName:6c2725fc-71ce-42c4-a633-a6bd1f16f58f nodeName:}" failed. No retries permitted until 2026-07-13 15:32:16.555040865 +0000 UTC m=+2362.752193804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-maas-api-tls") pod "maas-api-e2e-shared-a-88b181-965b9ff87-4xs24" (UID: "6c2725fc-71ce-42c4-a633-a6bd1f16f58f") : secret "maas-api-e2e-shared-a-88b181-serving-cert" not found
Jul 13 15:32:16.055493 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:16.055468    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-88b181-965b9ff87-4xs24\" (UID: \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:32:16.063582 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:16.063556    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqkj8\" (UniqueName: \"kubernetes.io/projected/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-kube-api-access-kqkj8\") pod \"maas-api-e2e-shared-a-88b181-965b9ff87-4xs24\" (UID: \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:32:16.558784 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:16.558748    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-maas-api-tls\") pod \"maas-api-e2e-shared-a-88b181-965b9ff87-4xs24\" (UID: \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:32:16.561446 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:16.561412    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-maas-api-tls\") pod \"maas-api-e2e-shared-a-88b181-965b9ff87-4xs24\" (UID: \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:32:16.688177 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:16.688141    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:32:16.862740 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:16.862715    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"]
Jul 13 15:32:16.864097 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:32:16.864071    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2725fc_71ce_42c4_a633_a6bd1f16f58f.slice/crio-422f51b2b214c4f83b9677f0bd391d7819d8dfb84b54a641c9de235454038ab1 WatchSource:0}: Error finding container 422f51b2b214c4f83b9677f0bd391d7819d8dfb84b54a641c9de235454038ab1: Status 404 returned error can't find the container with id 422f51b2b214c4f83b9677f0bd391d7819d8dfb84b54a641c9de235454038ab1
Jul 13 15:32:17.121413 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:17.121320    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24" event={"ID":"6c2725fc-71ce-42c4-a633-a6bd1f16f58f","Type":"ContainerStarted","Data":"422f51b2b214c4f83b9677f0bd391d7819d8dfb84b54a641c9de235454038ab1"}
Jul 13 15:32:18.127424 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:18.127385    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24" event={"ID":"6c2725fc-71ce-42c4-a633-a6bd1f16f58f","Type":"ContainerStarted","Data":"dc1fc8cee56e032c27495c5153e7b3b91caaae0bcd3f329276cbb3e03b9def44"}
Jul 13 15:32:18.127943 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:18.127547    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:32:18.146121 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:18.146076    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24" podStartSLOduration=2.790228681 podStartE2EDuration="3.146058381s" totalImagesPullingTime="355.8297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:32:18.145149347 +0000 UTC m=+2364.342302296" watchObservedRunningTime="2026-07-13 15:32:18.146058381 +0000 UTC m=+2364.343211329"
Jul 13 15:32:24.138281 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.138253    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:32:24.399582 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.399551    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"]
Jul 13 15:32:24.403827 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.403806    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:32:24.406606 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.406581    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-450dce-serving-cert\""
Jul 13 15:32:24.412527 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.412505    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"]
Jul 13 15:32:24.529825 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.529783    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083b6cd3-27c6-4d72-b052-9acbb035d16a-maas-api-tls\") pod \"maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t\" (UID: \"083b6cd3-27c6-4d72-b052-9acbb035d16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:32:24.530054 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.529854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p96ll\" (UniqueName: \"kubernetes.io/projected/083b6cd3-27c6-4d72-b052-9acbb035d16a-kube-api-access-p96ll\") pod \"maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t\" (UID: \"083b6cd3-27c6-4d72-b052-9acbb035d16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:32:24.530054 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.529925    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083b6cd3-27c6-4d72-b052-9acbb035d16a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t\" (UID: \"083b6cd3-27c6-4d72-b052-9acbb035d16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:32:24.631070 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.631034    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083b6cd3-27c6-4d72-b052-9acbb035d16a-maas-api-tls\") pod \"maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t\" (UID: \"083b6cd3-27c6-4d72-b052-9acbb035d16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:32:24.631203 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.631093    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p96ll\" (UniqueName: \"kubernetes.io/projected/083b6cd3-27c6-4d72-b052-9acbb035d16a-kube-api-access-p96ll\") pod \"maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t\" (UID: \"083b6cd3-27c6-4d72-b052-9acbb035d16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:32:24.631203 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.631125    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083b6cd3-27c6-4d72-b052-9acbb035d16a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t\" (UID: \"083b6cd3-27c6-4d72-b052-9acbb035d16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:32:24.631941 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.631879    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083b6cd3-27c6-4d72-b052-9acbb035d16a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t\" (UID: \"083b6cd3-27c6-4d72-b052-9acbb035d16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:32:24.633599 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.633578    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083b6cd3-27c6-4d72-b052-9acbb035d16a-maas-api-tls\") pod \"maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t\" (UID: \"083b6cd3-27c6-4d72-b052-9acbb035d16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:32:24.645244 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.645215    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p96ll\" (UniqueName: \"kubernetes.io/projected/083b6cd3-27c6-4d72-b052-9acbb035d16a-kube-api-access-p96ll\") pod \"maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t\" (UID: \"083b6cd3-27c6-4d72-b052-9acbb035d16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:32:24.715567 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.715455    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:32:24.864480 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:24.864460    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"]
Jul 13 15:32:24.866226 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:32:24.866200    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083b6cd3_27c6_4d72_b052_9acbb035d16a.slice/crio-c6ddbb981e0f0e5d1fb8567b2a93d299c7e544a46e30b3acf6d50eb56a47d189 WatchSource:0}: Error finding container c6ddbb981e0f0e5d1fb8567b2a93d299c7e544a46e30b3acf6d50eb56a47d189: Status 404 returned error can't find the container with id c6ddbb981e0f0e5d1fb8567b2a93d299c7e544a46e30b3acf6d50eb56a47d189
Jul 13 15:32:25.160574 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:25.160529    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t" event={"ID":"083b6cd3-27c6-4d72-b052-9acbb035d16a","Type":"ContainerStarted","Data":"c6ddbb981e0f0e5d1fb8567b2a93d299c7e544a46e30b3acf6d50eb56a47d189"}
Jul 13 15:32:26.166782 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:26.166745    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t" event={"ID":"083b6cd3-27c6-4d72-b052-9acbb035d16a","Type":"ContainerStarted","Data":"8575178a35fdd615c34d8c0bc8b9395554a246ec8adbb90982993b6ade45f888"}
Jul 13 15:32:26.167200 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:26.166887    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:32:26.184342 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:26.184291    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t" podStartSLOduration=1.798876306 podStartE2EDuration="2.18426912s" totalImagesPullingTime="385.392814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:32:26.183266872 +0000 UTC m=+2372.380419819" watchObservedRunningTime="2026-07-13 15:32:26.18426912 +0000 UTC m=+2372.381422072"
Jul 13 15:32:32.177798 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:32.177767    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:32:36.993324 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:36.993281    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"]
Jul 13 15:32:36.997142 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:36.997123    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:36.999548 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:36.999529    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-88b181\"/\"kube-root-ca.crt\""
Jul 13 15:32:37.000588 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.000564    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-88b181\"/\"auth-test-model-88b181-kserve-self-signed-certs\""
Jul 13 15:32:37.000721 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.000604    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-88b181\"/\"default-dockercfg-wlvgh\""
Jul 13 15:32:37.000721 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.000569    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-88b181\"/\"openshift-service-ca.crt\""
Jul 13 15:32:37.005416 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.005393    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"]
Jul 13 15:32:37.033304 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.033275    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-dshm\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.033424 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.033310    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-home\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.033424 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.033335    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-tls-certs\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.033498 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.033424    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-tmp-dir\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.033498 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.033454    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rddjz\" (UniqueName: \"kubernetes.io/projected/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-kube-api-access-rddjz\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.033498 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.033474    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-model-cache\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.134937 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.134904    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-dshm\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.134937 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.134938    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-home\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.135154 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.134956    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-tls-certs\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.135154 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.135096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-tmp-dir\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.135154 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.135130    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rddjz\" (UniqueName: \"kubernetes.io/projected/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-kube-api-access-rddjz\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.135302 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.135160    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-model-cache\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.135455 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.135426    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-tmp-dir\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.135559 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.135473    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-home\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.135605 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.135582    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-model-cache\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.137401 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.137372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-dshm\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.137585 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.137566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-tls-certs\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.143468 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.143445    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rddjz\" (UniqueName: \"kubernetes.io/projected/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-kube-api-access-rddjz\") pod \"auth-test-model-88b181-kserve-7885f8db67-mlvx7\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") " pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.308349 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.308272    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:37.442212 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:37.442178    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"]
Jul 13 15:32:37.445385 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:32:37.445358    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5ff4168_1c12_4dbe_881b_f9bdb0be8a6d.slice/crio-2eade2478c848be8d995c27546e9d88e6bf1937956fabfe3a7da0696efd6e9ee WatchSource:0}: Error finding container 2eade2478c848be8d995c27546e9d88e6bf1937956fabfe3a7da0696efd6e9ee: Status 404 returned error can't find the container with id 2eade2478c848be8d995c27546e9d88e6bf1937956fabfe3a7da0696efd6e9ee
Jul 13 15:32:38.224103 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:38.224064    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7" event={"ID":"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d","Type":"ContainerStarted","Data":"623e029a6e8ea9a95eb8cb1bf1232aab5d96833ba644b711e087db49d4747289"}
Jul 13 15:32:38.224103 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:38.224108    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7" event={"ID":"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d","Type":"ContainerStarted","Data":"2eade2478c848be8d995c27546e9d88e6bf1937956fabfe3a7da0696efd6e9ee"}
Jul 13 15:32:38.241446 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:38.241401    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7" podStartSLOduration=2.241384369 podStartE2EDuration="2.241384369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:32:38.23962785 +0000 UTC m=+2384.436780798" watchObservedRunningTime="2026-07-13 15:32:38.241384369 +0000 UTC m=+2384.438537316"
Jul 13 15:32:47.308570 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:47.308533    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:47.308987 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:47.308589    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:47.321095 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:47.321072    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:48.281387 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:48.281359    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:32:54.390579 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:54.390471    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:32:54.404812 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:32:54.392085    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:33:08.392070 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.392040    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"]
Jul 13 15:33:08.395669 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.395652    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.398453 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.398434    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-450dce\"/\"default-dockercfg-hlpkm\""
Jul 13 15:33:08.398550 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.398457    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-450dce\"/\"kube-root-ca.crt\""
Jul 13 15:33:08.399168 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.399150    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-450dce\"/\"openshift-service-ca.crt\""
Jul 13 15:33:08.399265 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.399247    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-450dce\"/\"auth-test-model-450dce-kserve-self-signed-certs\""
Jul 13 15:33:08.406683 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.406658    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"]
Jul 13 15:33:08.517269 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.517240    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-model-cache\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.517457 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.517279    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-tmp-dir\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.517457 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.517305    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-dshm\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.517457 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.517384    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecca273-f3ab-4502-b292-f44313a1a1e6-tls-certs\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.517457 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.517451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-home\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.517601 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.517492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxvh\" (UniqueName: \"kubernetes.io/projected/6ecca273-f3ab-4502-b292-f44313a1a1e6-kube-api-access-7jxvh\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.618128 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.618089    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxvh\" (UniqueName: \"kubernetes.io/projected/6ecca273-f3ab-4502-b292-f44313a1a1e6-kube-api-access-7jxvh\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.618128 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.618138    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-model-cache\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.618378 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.618157    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-tmp-dir\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.618378 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.618185    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-dshm\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.618378 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.618226    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecca273-f3ab-4502-b292-f44313a1a1e6-tls-certs\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.618378 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.618284    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-home\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.618644 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.618619    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-model-cache\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.618644 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.618631    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-tmp-dir\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.618788 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.618736    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-home\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.620557 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.620536    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-dshm\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.620889 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.620871    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecca273-f3ab-4502-b292-f44313a1a1e6-tls-certs\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.627944 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.627920    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxvh\" (UniqueName: \"kubernetes.io/projected/6ecca273-f3ab-4502-b292-f44313a1a1e6-kube-api-access-7jxvh\") pod \"auth-test-model-450dce-kserve-5487f8d58c-n7zdb\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") " pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.704888 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.704802    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:08.843419 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:08.843383    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"]
Jul 13 15:33:09.373661 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:09.373622    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb" event={"ID":"6ecca273-f3ab-4502-b292-f44313a1a1e6","Type":"ContainerStarted","Data":"1a2ff7fe4fe9e1705f334a237c757d402e65ecc0414e5af769f447898e5a65c4"}
Jul 13 15:33:09.373661 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:09.373669    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb" event={"ID":"6ecca273-f3ab-4502-b292-f44313a1a1e6","Type":"ContainerStarted","Data":"3c7b06196b5a7b6be7c1c3f72fcfbe7f8524ac91e71d6fe8cd9932edeecda154"}
Jul 13 15:33:09.393166 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:09.393116    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb" podStartSLOduration=1.393096777 podStartE2EDuration="1.393096777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:33:09.390905452 +0000 UTC m=+2415.588058400" watchObservedRunningTime="2026-07-13 15:33:09.393096777 +0000 UTC m=+2415.590249724"
Jul 13 15:33:18.705289 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:18.705190    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:18.705289 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:18.705240    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:18.718031 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:18.718005    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:33:19.433091 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:33:19.433063    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:35:17.836500 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:17.836413    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"]
Jul 13 15:35:18.812333 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:18.812292    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"]
Jul 13 15:35:32.857112 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:32.857070    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7" podUID="e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d" containerName="main" containerID="cri-o://623e029a6e8ea9a95eb8cb1bf1232aab5d96833ba644b711e087db49d4747289" gracePeriod=45
Jul 13 15:35:33.066720 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.066668    2564 generic.go:356] "Generic (PLEG): container finished" podID="e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d" containerID="623e029a6e8ea9a95eb8cb1bf1232aab5d96833ba644b711e087db49d4747289" exitCode=0
Jul 13 15:35:33.066872 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.066749    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7" event={"ID":"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d","Type":"ContainerDied","Data":"623e029a6e8ea9a95eb8cb1bf1232aab5d96833ba644b711e087db49d4747289"}
Jul 13 15:35:33.108088 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.108029    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:35:33.244937 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.244909    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-home\") pod \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") "
Jul 13 15:35:33.245147 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.244964    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rddjz\" (UniqueName: \"kubernetes.io/projected/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-kube-api-access-rddjz\") pod \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") "
Jul 13 15:35:33.245147 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.244994    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-dshm\") pod \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") "
Jul 13 15:35:33.245147 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.245053    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-tls-certs\") pod \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") "
Jul 13 15:35:33.245147 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.245077    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-tmp-dir\") pod \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") "
Jul 13 15:35:33.245406 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.245152    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-model-cache\") pod \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\" (UID: \"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d\") "
Jul 13 15:35:33.245406 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.245225    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-home" (OuterVolumeSpecName: "home") pod "e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d" (UID: "e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:35:33.245508 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.245418    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d" (UID: "e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:35:33.245508 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.245435    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d" (UID: "e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:35:33.245508 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.245431    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:35:33.247383 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.247358    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-dshm" (OuterVolumeSpecName: "dshm") pod "e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d" (UID: "e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:35:33.247491 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.247461    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-kube-api-access-rddjz" (OuterVolumeSpecName: "kube-api-access-rddjz") pod "e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d" (UID: "e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d"). InnerVolumeSpecName "kube-api-access-rddjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:35:33.247536 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.247485    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d" (UID: "e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:35:33.346585 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.346546    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rddjz\" (UniqueName: \"kubernetes.io/projected/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-kube-api-access-rddjz\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:35:33.346585 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.346575    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:35:33.346585 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.346586    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:35:33.346846 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.346615    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:35:33.346846 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.346623    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:35:33.832599 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:33.832524    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb" podUID="6ecca273-f3ab-4502-b292-f44313a1a1e6" containerName="main" containerID="cri-o://1a2ff7fe4fe9e1705f334a237c757d402e65ecc0414e5af769f447898e5a65c4" gracePeriod=45
Jul 13 15:35:34.072592 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.072565    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"
Jul 13 15:35:34.073026 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.072591    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7" event={"ID":"e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d","Type":"ContainerDied","Data":"2eade2478c848be8d995c27546e9d88e6bf1937956fabfe3a7da0696efd6e9ee"}
Jul 13 15:35:34.073026 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.072653    2564 scope.go:117] "RemoveContainer" containerID="623e029a6e8ea9a95eb8cb1bf1232aab5d96833ba644b711e087db49d4747289"
Jul 13 15:35:34.082186 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.076153    2564 generic.go:356] "Generic (PLEG): container finished" podID="6ecca273-f3ab-4502-b292-f44313a1a1e6" containerID="1a2ff7fe4fe9e1705f334a237c757d402e65ecc0414e5af769f447898e5a65c4" exitCode=0
Jul 13 15:35:34.082186 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.076323    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb" event={"ID":"6ecca273-f3ab-4502-b292-f44313a1a1e6","Type":"ContainerDied","Data":"1a2ff7fe4fe9e1705f334a237c757d402e65ecc0414e5af769f447898e5a65c4"}
Jul 13 15:35:34.095838 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.095818    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:35:34.109324 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.109299    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"]
Jul 13 15:35:34.116226 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.116203    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/auth-test-model-88b181-kserve-7885f8db67-mlvx7"]
Jul 13 15:35:34.254939 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.254890    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-tmp-dir\") pod \"6ecca273-f3ab-4502-b292-f44313a1a1e6\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") "
Jul 13 15:35:34.255142 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.254965    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-model-cache\") pod \"6ecca273-f3ab-4502-b292-f44313a1a1e6\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") "
Jul 13 15:35:34.255142 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.254994    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-dshm\") pod \"6ecca273-f3ab-4502-b292-f44313a1a1e6\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") "
Jul 13 15:35:34.255142 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.255032    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-home\") pod \"6ecca273-f3ab-4502-b292-f44313a1a1e6\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") "
Jul 13 15:35:34.255142 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.255062    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecca273-f3ab-4502-b292-f44313a1a1e6-tls-certs\") pod \"6ecca273-f3ab-4502-b292-f44313a1a1e6\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") "
Jul 13 15:35:34.255142 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.255094    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jxvh\" (UniqueName: \"kubernetes.io/projected/6ecca273-f3ab-4502-b292-f44313a1a1e6-kube-api-access-7jxvh\") pod \"6ecca273-f3ab-4502-b292-f44313a1a1e6\" (UID: \"6ecca273-f3ab-4502-b292-f44313a1a1e6\") "
Jul 13 15:35:34.255410 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.255152    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ecca273-f3ab-4502-b292-f44313a1a1e6" (UID: "6ecca273-f3ab-4502-b292-f44313a1a1e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:35:34.255410 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.255281    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ecca273-f3ab-4502-b292-f44313a1a1e6" (UID: "6ecca273-f3ab-4502-b292-f44313a1a1e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:35:34.255410 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.255297    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-home" (OuterVolumeSpecName: "home") pod "6ecca273-f3ab-4502-b292-f44313a1a1e6" (UID: "6ecca273-f3ab-4502-b292-f44313a1a1e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:35:34.255410 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.255409    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:35:34.255562 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.255425    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:35:34.255562 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.255440    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:35:34.257405 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.257381    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ecca273-f3ab-4502-b292-f44313a1a1e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ecca273-f3ab-4502-b292-f44313a1a1e6" (UID: "6ecca273-f3ab-4502-b292-f44313a1a1e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:35:34.257523 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.257435    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-dshm" (OuterVolumeSpecName: "dshm") pod "6ecca273-f3ab-4502-b292-f44313a1a1e6" (UID: "6ecca273-f3ab-4502-b292-f44313a1a1e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:35:34.257523 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.257445    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecca273-f3ab-4502-b292-f44313a1a1e6-kube-api-access-7jxvh" (OuterVolumeSpecName: "kube-api-access-7jxvh") pod "6ecca273-f3ab-4502-b292-f44313a1a1e6" (UID: "6ecca273-f3ab-4502-b292-f44313a1a1e6"). InnerVolumeSpecName "kube-api-access-7jxvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:35:34.316623 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.316591    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d" path="/var/lib/kubelet/pods/e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d/volumes"
Jul 13 15:35:34.356681 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.356620    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecca273-f3ab-4502-b292-f44313a1a1e6-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:35:34.356681 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.356645    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jxvh\" (UniqueName: \"kubernetes.io/projected/6ecca273-f3ab-4502-b292-f44313a1a1e6-kube-api-access-7jxvh\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:35:34.356681 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:34.356654    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecca273-f3ab-4502-b292-f44313a1a1e6-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:35:35.083288 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:35.083263    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"
Jul 13 15:35:35.083790 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:35.083308    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb" event={"ID":"6ecca273-f3ab-4502-b292-f44313a1a1e6","Type":"ContainerDied","Data":"3c7b06196b5a7b6be7c1c3f72fcfbe7f8524ac91e71d6fe8cd9932edeecda154"}
Jul 13 15:35:35.083790 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:35.083357    2564 scope.go:117] "RemoveContainer" containerID="1a2ff7fe4fe9e1705f334a237c757d402e65ecc0414e5af769f447898e5a65c4"
Jul 13 15:35:35.108944 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:35.108915    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"]
Jul 13 15:35:35.112230 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:35.112207    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/auth-test-model-450dce-kserve-5487f8d58c-n7zdb"]
Jul 13 15:35:36.316938 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:36.316906    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ecca273-f3ab-4502-b292-f44313a1a1e6" path="/var/lib/kubelet/pods/6ecca273-f3ab-4502-b292-f44313a1a1e6/volumes"
Jul 13 15:35:41.793344 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.793305    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"]
Jul 13 15:35:41.793758 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.793690    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d" containerName="main"
Jul 13 15:35:41.793758 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.793718    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d" containerName="main"
Jul 13 15:35:41.793758 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.793744    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ecca273-f3ab-4502-b292-f44313a1a1e6" containerName="main"
Jul 13 15:35:41.793758 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.793751    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ecca273-f3ab-4502-b292-f44313a1a1e6" containerName="main"
Jul 13 15:35:41.793967 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.793818    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ecca273-f3ab-4502-b292-f44313a1a1e6" containerName="main"
Jul 13 15:35:41.793967 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.793828    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5ff4168-1c12-4dbe-881b-f9bdb0be8a6d" containerName="main"
Jul 13 15:35:41.798271 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.798253    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.800950 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.800926    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-88b181\"/\"kube-root-ca.crt\""
Jul 13 15:35:41.801837 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.801814    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-88b181\"/\"openshift-service-ca.crt\""
Jul 13 15:35:41.801947 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.801930    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-88b181\"/\"sub-test-model-88b181-kserve-self-signed-certs\""
Jul 13 15:35:41.802024 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.801989    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-88b181\"/\"default-dockercfg-wlvgh\""
Jul 13 15:35:41.807932 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.807909    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"]
Jul 13 15:35:41.824540 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.824514    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-model-cache\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.824679 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.824584    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-home\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.824679 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.824623    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffacbda-8fdf-4085-8316-8bf63af79a6d-tls-certs\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.824830 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.824716    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4psf\" (UniqueName: \"kubernetes.io/projected/4ffacbda-8fdf-4085-8316-8bf63af79a6d-kube-api-access-h4psf\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.824830 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.824754    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-tmp-dir\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.824830 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.824803    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-dshm\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.926215 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.926174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-home\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.926215 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.926215    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffacbda-8fdf-4085-8316-8bf63af79a6d-tls-certs\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.926460 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.926252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4psf\" (UniqueName: \"kubernetes.io/projected/4ffacbda-8fdf-4085-8316-8bf63af79a6d-kube-api-access-h4psf\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.926460 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.926278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-tmp-dir\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.926460 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.926299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-dshm\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.926460 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.926331    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-model-cache\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.926675 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.926650    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-home\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.926769 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.926681    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-model-cache\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.926769 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.926657    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-tmp-dir\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.928686 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.928667    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-dshm\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.928993 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.928974    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffacbda-8fdf-4085-8316-8bf63af79a6d-tls-certs\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:41.936558 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:41.936530    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4psf\" (UniqueName: \"kubernetes.io/projected/4ffacbda-8fdf-4085-8316-8bf63af79a6d-kube-api-access-h4psf\") pod \"sub-test-model-88b181-kserve-76675d4b75-6thx5\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") " pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:42.108008 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:42.107919    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:42.254942 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:42.254922    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"]
Jul 13 15:35:42.256807 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:35:42.256779    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ffacbda_8fdf_4085_8316_8bf63af79a6d.slice/crio-98404c616b40e12e574fbce83d1e349edc91d877c8ab7024aeb5f48e1d66d923 WatchSource:0}: Error finding container 98404c616b40e12e574fbce83d1e349edc91d877c8ab7024aeb5f48e1d66d923: Status 404 returned error can't find the container with id 98404c616b40e12e574fbce83d1e349edc91d877c8ab7024aeb5f48e1d66d923
Jul 13 15:35:43.125004 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:43.124967    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5" event={"ID":"4ffacbda-8fdf-4085-8316-8bf63af79a6d","Type":"ContainerStarted","Data":"9d1885838dd051f14ec58d547bd2261e7656a65ae841cb916ea45a894dd880ac"}
Jul 13 15:35:43.125004 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:43.125008    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5" event={"ID":"4ffacbda-8fdf-4085-8316-8bf63af79a6d","Type":"ContainerStarted","Data":"98404c616b40e12e574fbce83d1e349edc91d877c8ab7024aeb5f48e1d66d923"}
Jul 13 15:35:43.146171 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:43.146126    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5" podStartSLOduration=2.146111197 podStartE2EDuration="2.146111197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:35:43.145309855 +0000 UTC m=+2569.342462803" watchObservedRunningTime="2026-07-13 15:35:43.146111197 +0000 UTC m=+2569.343264144"
Jul 13 15:35:52.108351 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:52.108315    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:52.108894 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:52.108612    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:52.121283 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:52.121258    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:35:52.195325 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:35:52.195298    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:36:08.588336 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.588303    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"]
Jul 13 15:36:08.591599 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.591574    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.594409 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.594380    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-450dce\"/\"kube-root-ca.crt\""
Jul 13 15:36:08.595298 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.595270    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-450dce\"/\"openshift-service-ca.crt\""
Jul 13 15:36:08.595630 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.595355    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-450dce\"/\"default-dockercfg-hlpkm\""
Jul 13 15:36:08.595630 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.595368    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-450dce\"/\"sub-test-model-450dce-kserve-self-signed-certs\""
Jul 13 15:36:08.603520 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.603499    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"]
Jul 13 15:36:08.669926 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.669890    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7m9h\" (UniqueName: \"kubernetes.io/projected/f6532531-d01b-4c0f-9fd5-046342f86121-kube-api-access-g7m9h\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.670141 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.669949    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-dshm\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.670141 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.670019    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6532531-d01b-4c0f-9fd5-046342f86121-tls-certs\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.670141 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.670070    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-tmp-dir\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.670141 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.670101    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-model-cache\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.670141 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.670127    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-home\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.771285 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.771241    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-dshm\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.771483 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.771295    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6532531-d01b-4c0f-9fd5-046342f86121-tls-certs\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.771483 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.771320    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-tmp-dir\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.771483 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.771344    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-model-cache\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.771483 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.771368    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-home\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.771483 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.771443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7m9h\" (UniqueName: \"kubernetes.io/projected/f6532531-d01b-4c0f-9fd5-046342f86121-kube-api-access-g7m9h\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.771818 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.771784    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-model-cache\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.771818 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.771812    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-tmp-dir\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.771965 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.771862    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-home\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.773831 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.773807    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-dshm\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.774039 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.774024    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6532531-d01b-4c0f-9fd5-046342f86121-tls-certs\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.780337 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.780317    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7m9h\" (UniqueName: \"kubernetes.io/projected/f6532531-d01b-4c0f-9fd5-046342f86121-kube-api-access-g7m9h\") pod \"sub-test-model-450dce-kserve-559df5c45c-9pwwt\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") " pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:08.901776 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:08.901737    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:09.043185 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:09.043157    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"]
Jul 13 15:36:09.045573 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:36:09.045542    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6532531_d01b_4c0f_9fd5_046342f86121.slice/crio-7b9d974531a1da4bc0b7368d3db9c6c7524951493a9ee602817b272692e7e660 WatchSource:0}: Error finding container 7b9d974531a1da4bc0b7368d3db9c6c7524951493a9ee602817b272692e7e660: Status 404 returned error can't find the container with id 7b9d974531a1da4bc0b7368d3db9c6c7524951493a9ee602817b272692e7e660
Jul 13 15:36:09.260931 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:09.260841    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt" event={"ID":"f6532531-d01b-4c0f-9fd5-046342f86121","Type":"ContainerStarted","Data":"f04dcd4e62ea97ebb80845cb2214eac09f523a20cfaec3f38f1aad55d049fd78"}
Jul 13 15:36:09.260931 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:09.260886    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt" event={"ID":"f6532531-d01b-4c0f-9fd5-046342f86121","Type":"ContainerStarted","Data":"7b9d974531a1da4bc0b7368d3db9c6c7524951493a9ee602817b272692e7e660"}
Jul 13 15:36:09.279217 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:09.279158    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt" podStartSLOduration=1.279140855 podStartE2EDuration="1.279140855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:36:09.277626869 +0000 UTC m=+2595.474779818" watchObservedRunningTime="2026-07-13 15:36:09.279140855 +0000 UTC m=+2595.476293802"
Jul 13 15:36:18.901895 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:18.901817    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:18.901895 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:18.901873    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:18.914313 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:18.914285    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:36:19.322539 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:36:19.322453    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:37:01.116404 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:01.116370    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"]
Jul 13 15:37:02.014685 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:02.014650    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"]
Jul 13 15:37:16.136651 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.136608    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5" podUID="4ffacbda-8fdf-4085-8316-8bf63af79a6d" containerName="main" containerID="cri-o://9d1885838dd051f14ec58d547bd2261e7656a65ae841cb916ea45a894dd880ac" gracePeriod=45
Jul 13 15:37:16.397629 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.397602    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:37:16.492262 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.492223    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-model-cache\") pod \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") "
Jul 13 15:37:16.492433 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.492291    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4psf\" (UniqueName: \"kubernetes.io/projected/4ffacbda-8fdf-4085-8316-8bf63af79a6d-kube-api-access-h4psf\") pod \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") "
Jul 13 15:37:16.492433 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.492363    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-tmp-dir\") pod \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") "
Jul 13 15:37:16.492433 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.492413    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-home\") pod \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") "
Jul 13 15:37:16.492557 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.492439    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-dshm\") pod \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") "
Jul 13 15:37:16.492557 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.492479    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffacbda-8fdf-4085-8316-8bf63af79a6d-tls-certs\") pod \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\" (UID: \"4ffacbda-8fdf-4085-8316-8bf63af79a6d\") "
Jul 13 15:37:16.492665 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.492571    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ffacbda-8fdf-4085-8316-8bf63af79a6d" (UID: "4ffacbda-8fdf-4085-8316-8bf63af79a6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:37:16.492754 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.492726    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ffacbda-8fdf-4085-8316-8bf63af79a6d" (UID: "4ffacbda-8fdf-4085-8316-8bf63af79a6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:37:16.492821 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.492743    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-home" (OuterVolumeSpecName: "home") pod "4ffacbda-8fdf-4085-8316-8bf63af79a6d" (UID: "4ffacbda-8fdf-4085-8316-8bf63af79a6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:37:16.492882 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.492841    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:37:16.492882 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.492860    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:37:16.492882 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.492873    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:37:16.494712 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.494672    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ffacbda-8fdf-4085-8316-8bf63af79a6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ffacbda-8fdf-4085-8316-8bf63af79a6d" (UID: "4ffacbda-8fdf-4085-8316-8bf63af79a6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:37:16.494797 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.494738    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffacbda-8fdf-4085-8316-8bf63af79a6d-kube-api-access-h4psf" (OuterVolumeSpecName: "kube-api-access-h4psf") pod "4ffacbda-8fdf-4085-8316-8bf63af79a6d" (UID: "4ffacbda-8fdf-4085-8316-8bf63af79a6d"). InnerVolumeSpecName "kube-api-access-h4psf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:37:16.494797 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.494757    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-dshm" (OuterVolumeSpecName: "dshm") pod "4ffacbda-8fdf-4085-8316-8bf63af79a6d" (UID: "4ffacbda-8fdf-4085-8316-8bf63af79a6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:37:16.587777 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.587740    2564 generic.go:356] "Generic (PLEG): container finished" podID="4ffacbda-8fdf-4085-8316-8bf63af79a6d" containerID="9d1885838dd051f14ec58d547bd2261e7656a65ae841cb916ea45a894dd880ac" exitCode=0
Jul 13 15:37:16.587942 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.587786    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5" event={"ID":"4ffacbda-8fdf-4085-8316-8bf63af79a6d","Type":"ContainerDied","Data":"9d1885838dd051f14ec58d547bd2261e7656a65ae841cb916ea45a894dd880ac"}
Jul 13 15:37:16.587942 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.587821    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5" event={"ID":"4ffacbda-8fdf-4085-8316-8bf63af79a6d","Type":"ContainerDied","Data":"98404c616b40e12e574fbce83d1e349edc91d877c8ab7024aeb5f48e1d66d923"}
Jul 13 15:37:16.587942 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.587837    2564 scope.go:117] "RemoveContainer" containerID="9d1885838dd051f14ec58d547bd2261e7656a65ae841cb916ea45a894dd880ac"
Jul 13 15:37:16.587942 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.587793    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"
Jul 13 15:37:16.593366 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.593344    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ffacbda-8fdf-4085-8316-8bf63af79a6d-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:37:16.593366 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.593367    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ffacbda-8fdf-4085-8316-8bf63af79a6d-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:37:16.593500 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.593378    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4psf\" (UniqueName: \"kubernetes.io/projected/4ffacbda-8fdf-4085-8316-8bf63af79a6d-kube-api-access-h4psf\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:37:16.598037 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.598018    2564 scope.go:117] "RemoveContainer" containerID="9d1885838dd051f14ec58d547bd2261e7656a65ae841cb916ea45a894dd880ac"
Jul 13 15:37:16.598318 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:37:16.598277    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d1885838dd051f14ec58d547bd2261e7656a65ae841cb916ea45a894dd880ac\": container with ID starting with 9d1885838dd051f14ec58d547bd2261e7656a65ae841cb916ea45a894dd880ac not found: ID does not exist" containerID="9d1885838dd051f14ec58d547bd2261e7656a65ae841cb916ea45a894dd880ac"
Jul 13 15:37:16.598399 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.598309    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1885838dd051f14ec58d547bd2261e7656a65ae841cb916ea45a894dd880ac"} err="failed to get container status \"9d1885838dd051f14ec58d547bd2261e7656a65ae841cb916ea45a894dd880ac\": rpc error: code = NotFound desc = could not find container \"9d1885838dd051f14ec58d547bd2261e7656a65ae841cb916ea45a894dd880ac\": container with ID starting with 9d1885838dd051f14ec58d547bd2261e7656a65ae841cb916ea45a894dd880ac not found: ID does not exist"
Jul 13 15:37:16.617388 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.617362    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"]
Jul 13 15:37:16.622238 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:16.622215    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/sub-test-model-88b181-kserve-76675d4b75-6thx5"]
Jul 13 15:37:17.034843 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.034800    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt" podUID="f6532531-d01b-4c0f-9fd5-046342f86121" containerName="main" containerID="cri-o://f04dcd4e62ea97ebb80845cb2214eac09f523a20cfaec3f38f1aad55d049fd78" gracePeriod=45
Jul 13 15:37:17.291667 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.291599    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:37:17.400597 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.400563    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-model-cache\") pod \"f6532531-d01b-4c0f-9fd5-046342f86121\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") "
Jul 13 15:37:17.400791 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.400638    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7m9h\" (UniqueName: \"kubernetes.io/projected/f6532531-d01b-4c0f-9fd5-046342f86121-kube-api-access-g7m9h\") pod \"f6532531-d01b-4c0f-9fd5-046342f86121\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") "
Jul 13 15:37:17.400791 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.400664    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-dshm\") pod \"f6532531-d01b-4c0f-9fd5-046342f86121\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") "
Jul 13 15:37:17.400791 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.400728    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6532531-d01b-4c0f-9fd5-046342f86121-tls-certs\") pod \"f6532531-d01b-4c0f-9fd5-046342f86121\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") "
Jul 13 15:37:17.400791 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.400748    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-home\") pod \"f6532531-d01b-4c0f-9fd5-046342f86121\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") "
Jul 13 15:37:17.400791 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.400771    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-tmp-dir\") pod \"f6532531-d01b-4c0f-9fd5-046342f86121\" (UID: \"f6532531-d01b-4c0f-9fd5-046342f86121\") "
Jul 13 15:37:17.401059 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.400923    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6532531-d01b-4c0f-9fd5-046342f86121" (UID: "f6532531-d01b-4c0f-9fd5-046342f86121"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:37:17.401059 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.401055    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:37:17.401172 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.401071    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-home" (OuterVolumeSpecName: "home") pod "f6532531-d01b-4c0f-9fd5-046342f86121" (UID: "f6532531-d01b-4c0f-9fd5-046342f86121"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:37:17.401172 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.401142    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6532531-d01b-4c0f-9fd5-046342f86121" (UID: "f6532531-d01b-4c0f-9fd5-046342f86121"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:37:17.403050 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.403025    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-dshm" (OuterVolumeSpecName: "dshm") pod "f6532531-d01b-4c0f-9fd5-046342f86121" (UID: "f6532531-d01b-4c0f-9fd5-046342f86121"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:37:17.403050 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.403043    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6532531-d01b-4c0f-9fd5-046342f86121-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6532531-d01b-4c0f-9fd5-046342f86121" (UID: "f6532531-d01b-4c0f-9fd5-046342f86121"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:37:17.403259 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.403244    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6532531-d01b-4c0f-9fd5-046342f86121-kube-api-access-g7m9h" (OuterVolumeSpecName: "kube-api-access-g7m9h") pod "f6532531-d01b-4c0f-9fd5-046342f86121" (UID: "f6532531-d01b-4c0f-9fd5-046342f86121"). InnerVolumeSpecName "kube-api-access-g7m9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:37:17.502047 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.502003    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6532531-d01b-4c0f-9fd5-046342f86121-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:37:17.502047 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.502043    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:37:17.502047 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.502057    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:37:17.502293 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.502071    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7m9h\" (UniqueName: \"kubernetes.io/projected/f6532531-d01b-4c0f-9fd5-046342f86121-kube-api-access-g7m9h\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:37:17.502293 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.502084    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6532531-d01b-4c0f-9fd5-046342f86121-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:37:17.596301 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.596203    2564 generic.go:356] "Generic (PLEG): container finished" podID="f6532531-d01b-4c0f-9fd5-046342f86121" containerID="f04dcd4e62ea97ebb80845cb2214eac09f523a20cfaec3f38f1aad55d049fd78" exitCode=0
Jul 13 15:37:17.596301 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.596253    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"
Jul 13 15:37:17.596509 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.596293    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt" event={"ID":"f6532531-d01b-4c0f-9fd5-046342f86121","Type":"ContainerDied","Data":"f04dcd4e62ea97ebb80845cb2214eac09f523a20cfaec3f38f1aad55d049fd78"}
Jul 13 15:37:17.596509 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.596339    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt" event={"ID":"f6532531-d01b-4c0f-9fd5-046342f86121","Type":"ContainerDied","Data":"7b9d974531a1da4bc0b7368d3db9c6c7524951493a9ee602817b272692e7e660"}
Jul 13 15:37:17.596509 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.596361    2564 scope.go:117] "RemoveContainer" containerID="f04dcd4e62ea97ebb80845cb2214eac09f523a20cfaec3f38f1aad55d049fd78"
Jul 13 15:37:17.605999 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.605976    2564 scope.go:117] "RemoveContainer" containerID="f04dcd4e62ea97ebb80845cb2214eac09f523a20cfaec3f38f1aad55d049fd78"
Jul 13 15:37:17.606297 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:37:17.606269    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04dcd4e62ea97ebb80845cb2214eac09f523a20cfaec3f38f1aad55d049fd78\": container with ID starting with f04dcd4e62ea97ebb80845cb2214eac09f523a20cfaec3f38f1aad55d049fd78 not found: ID does not exist" containerID="f04dcd4e62ea97ebb80845cb2214eac09f523a20cfaec3f38f1aad55d049fd78"
Jul 13 15:37:17.606361 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.606308    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04dcd4e62ea97ebb80845cb2214eac09f523a20cfaec3f38f1aad55d049fd78"} err="failed to get container status \"f04dcd4e62ea97ebb80845cb2214eac09f523a20cfaec3f38f1aad55d049fd78\": rpc error: code = NotFound desc = could not find container \"f04dcd4e62ea97ebb80845cb2214eac09f523a20cfaec3f38f1aad55d049fd78\": container with ID starting with f04dcd4e62ea97ebb80845cb2214eac09f523a20cfaec3f38f1aad55d049fd78 not found: ID does not exist"
Jul 13 15:37:17.620675 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.620629    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"]
Jul 13 15:37:17.624032 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:17.624001    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/sub-test-model-450dce-kserve-559df5c45c-9pwwt"]
Jul 13 15:37:18.316688 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:18.316655    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ffacbda-8fdf-4085-8316-8bf63af79a6d" path="/var/lib/kubelet/pods/4ffacbda-8fdf-4085-8316-8bf63af79a6d/volumes"
Jul 13 15:37:18.317121 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:18.317104    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6532531-d01b-4c0f-9fd5-046342f86121" path="/var/lib/kubelet/pods/f6532531-d01b-4c0f-9fd5-046342f86121/volumes"
Jul 13 15:37:24.391016 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.390982    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"]
Jul 13 15:37:24.391457 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.391331    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6532531-d01b-4c0f-9fd5-046342f86121" containerName="main"
Jul 13 15:37:24.391457 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.391341    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6532531-d01b-4c0f-9fd5-046342f86121" containerName="main"
Jul 13 15:37:24.391457 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.391364    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ffacbda-8fdf-4085-8316-8bf63af79a6d" containerName="main"
Jul 13 15:37:24.391457 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.391369    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ffacbda-8fdf-4085-8316-8bf63af79a6d" containerName="main"
Jul 13 15:37:24.391457 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.391417    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6532531-d01b-4c0f-9fd5-046342f86121" containerName="main"
Jul 13 15:37:24.391457 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.391427    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ffacbda-8fdf-4085-8316-8bf63af79a6d" containerName="main"
Jul 13 15:37:24.395451 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.395432    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.398832 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.398807    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-88b181\"/\"kube-root-ca.crt\""
Jul 13 15:37:24.398832 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.398811    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-88b181\"/\"openshift-service-ca.crt\""
Jul 13 15:37:24.399004 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.398845    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-88b181\"/\"rate-test-model-88b181-kserve-self-signed-certs\""
Jul 13 15:37:24.399004 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.398936    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-88b181\"/\"default-dockercfg-wlvgh\""
Jul 13 15:37:24.402679 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.402657    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"]
Jul 13 15:37:24.463182 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.463150    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-model-cache\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.463374 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.463194    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-tls-certs\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.463374 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.463317    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfbt\" (UniqueName: \"kubernetes.io/projected/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-kube-api-access-9sfbt\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.463374 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.463360    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-home\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.463540 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.463408    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-dshm\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.463540 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.463446    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-tmp-dir\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.564144 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.564104    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-tls-certs\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.564344 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.564174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfbt\" (UniqueName: \"kubernetes.io/projected/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-kube-api-access-9sfbt\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.564344 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.564198    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-home\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.564344 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.564216    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-dshm\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.564344 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.564243    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-tmp-dir\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.564344 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.564283    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-model-cache\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.564770 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.564734    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-home\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.564770 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.564750    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-model-cache\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.564944 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.564779    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-tmp-dir\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.566612 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.566594    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-dshm\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.567010 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.566988    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-tls-certs\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.573243 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.573207    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfbt\" (UniqueName: \"kubernetes.io/projected/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-kube-api-access-9sfbt\") pod \"rate-test-model-88b181-kserve-7c59c4844c-6zj4t\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") " pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.705374 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.705339    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:24.842433 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:24.842333    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"]
Jul 13 15:37:24.847682 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:37:24.846506    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf322cdb7_4ae0_4cfc_b7f8_9c8d1868d96e.slice/crio-2fe9a1117cbb0ae883198083b019260ec496ea27e03cb9c599cad1110298ca1c WatchSource:0}: Error finding container 2fe9a1117cbb0ae883198083b019260ec496ea27e03cb9c599cad1110298ca1c: Status 404 returned error can't find the container with id 2fe9a1117cbb0ae883198083b019260ec496ea27e03cb9c599cad1110298ca1c
Jul 13 15:37:25.636323 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:25.636284    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t" event={"ID":"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e","Type":"ContainerStarted","Data":"22f462ddbdb766e9a5aa6514623666aaa667cbdf0c5a1d9df691707288cac131"}
Jul 13 15:37:25.636323 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:25.636328    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t" event={"ID":"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e","Type":"ContainerStarted","Data":"2fe9a1117cbb0ae883198083b019260ec496ea27e03cb9c599cad1110298ca1c"}
Jul 13 15:37:25.660641 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:25.660580    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t" podStartSLOduration=1.660564529 podStartE2EDuration="1.660564529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:37:25.658656604 +0000 UTC m=+2671.855809552" watchObservedRunningTime="2026-07-13 15:37:25.660564529 +0000 UTC m=+2671.857717478"
Jul 13 15:37:34.706232 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:34.706193    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:34.706758 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:34.706429    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:34.719457 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:34.719431    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:35.694238 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:35.694205    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:37:46.378759 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.378673    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"]
Jul 13 15:37:46.382031 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.382007    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.385393 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.385371    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-450dce\"/\"kube-root-ca.crt\""
Jul 13 15:37:46.385393 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.385390    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-450dce\"/\"openshift-service-ca.crt\""
Jul 13 15:37:46.385595 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.385408    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-450dce\"/\"rate-test-model-450dce-kserve-self-signed-certs\""
Jul 13 15:37:46.385595 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.385380    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-450dce\"/\"default-dockercfg-hlpkm\""
Jul 13 15:37:46.395353 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.395330    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"]
Jul 13 15:37:46.561208 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.561172    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-dshm\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.561208 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.561207    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4gpg\" (UniqueName: \"kubernetes.io/projected/02576314-bdc4-43c9-8b5c-1e13cd9344b6-kube-api-access-v4gpg\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.561430 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.561245    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-home\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.561430 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.561289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-tmp-dir\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.561430 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.561328    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02576314-bdc4-43c9-8b5c-1e13cd9344b6-tls-certs\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.561430 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.561352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-model-cache\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.662780 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.662743    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02576314-bdc4-43c9-8b5c-1e13cd9344b6-tls-certs\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.662969 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.662789    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-model-cache\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.662969 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.662928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-dshm\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.662969 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.662960    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4gpg\" (UniqueName: \"kubernetes.io/projected/02576314-bdc4-43c9-8b5c-1e13cd9344b6-kube-api-access-v4gpg\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.663153 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.663096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-home\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.663153 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.663142    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-model-cache\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.663153 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.663146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-tmp-dir\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.663409 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.663385    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-home\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.663409 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.663397    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-tmp-dir\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.665229 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.665210    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-dshm\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.665416 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.665399    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02576314-bdc4-43c9-8b5c-1e13cd9344b6-tls-certs\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.671484 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.671453    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4gpg\" (UniqueName: \"kubernetes.io/projected/02576314-bdc4-43c9-8b5c-1e13cd9344b6-kube-api-access-v4gpg\") pod \"rate-test-model-450dce-kserve-8564467cbc-6mx6r\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") " pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.691769 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.691742    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:46.824120 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:46.824095    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"]
Jul 13 15:37:46.826353 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:37:46.826328    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02576314_bdc4_43c9_8b5c_1e13cd9344b6.slice/crio-a2c3e4817da11d577651c5985d346865e79fbbe7947a5d79c9bef15b4b25ffb5 WatchSource:0}: Error finding container a2c3e4817da11d577651c5985d346865e79fbbe7947a5d79c9bef15b4b25ffb5: Status 404 returned error can't find the container with id a2c3e4817da11d577651c5985d346865e79fbbe7947a5d79c9bef15b4b25ffb5
Jul 13 15:37:47.742115 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:47.742078    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r" event={"ID":"02576314-bdc4-43c9-8b5c-1e13cd9344b6","Type":"ContainerStarted","Data":"121ca3f3f760b90f405a88fb80762386b5671df973d5ab13486e1f417222e89c"}
Jul 13 15:37:47.742115 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:47.742118    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r" event={"ID":"02576314-bdc4-43c9-8b5c-1e13cd9344b6","Type":"ContainerStarted","Data":"a2c3e4817da11d577651c5985d346865e79fbbe7947a5d79c9bef15b4b25ffb5"}
Jul 13 15:37:47.759040 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:47.758991    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r" podStartSLOduration=1.758976461 podStartE2EDuration="1.758976461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:37:47.758338728 +0000 UTC m=+2693.955491675" watchObservedRunningTime="2026-07-13 15:37:47.758976461 +0000 UTC m=+2693.956129408"
Jul 13 15:37:54.420466 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:54.420437    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:37:54.422930 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:54.422913    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tfgpx_8ffb49a6-e992-4bd6-a0ee-2e9c41258047/ovn-acl-logging/0.log"
Jul 13 15:37:56.692190 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:56.692149    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:56.692616 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:56.692213    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:56.704796 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:56.704773    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:37:56.796486 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:37:56.796459    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:38:29.107446 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:29.107409    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"]
Jul 13 15:38:29.992542 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:29.992508    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"]
Jul 13 15:38:44.128127 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.128078    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t" podUID="f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e" containerName="main" containerID="cri-o://22f462ddbdb766e9a5aa6514623666aaa667cbdf0c5a1d9df691707288cac131" gracePeriod=45
Jul 13 15:38:44.380880 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.380824    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:38:44.479108 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.479079    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-model-cache\") pod \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") "
Jul 13 15:38:44.479284 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.479121    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sfbt\" (UniqueName: \"kubernetes.io/projected/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-kube-api-access-9sfbt\") pod \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") "
Jul 13 15:38:44.479284 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.479178    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-home\") pod \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") "
Jul 13 15:38:44.479284 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.479214    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-dshm\") pod \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") "
Jul 13 15:38:44.479284 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.479255    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-tls-certs\") pod \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") "
Jul 13 15:38:44.479501 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.479298    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-tmp-dir\") pod \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\" (UID: \"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e\") "
Jul 13 15:38:44.479501 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.479324    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e" (UID: "f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:38:44.479501 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.479484    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-home" (OuterVolumeSpecName: "home") pod "f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e" (UID: "f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:38:44.479634 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.479558    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:38:44.479634 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.479575    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:38:44.479634 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.479604    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e" (UID: "f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:38:44.481441 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.481418    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e" (UID: "f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:38:44.481920 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.481899    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-dshm" (OuterVolumeSpecName: "dshm") pod "f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e" (UID: "f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:38:44.482039 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.481916    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-kube-api-access-9sfbt" (OuterVolumeSpecName: "kube-api-access-9sfbt") pod "f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e" (UID: "f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e"). InnerVolumeSpecName "kube-api-access-9sfbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:38:44.580112 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.580081    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:38:44.580269 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.580129    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:38:44.580269 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.580138    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sfbt\" (UniqueName: \"kubernetes.io/projected/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-kube-api-access-9sfbt\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:38:44.580269 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:44.580147    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:38:45.013231 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.013166    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r" podUID="02576314-bdc4-43c9-8b5c-1e13cd9344b6" containerName="main" containerID="cri-o://121ca3f3f760b90f405a88fb80762386b5671df973d5ab13486e1f417222e89c" gracePeriod=45
Jul 13 15:38:45.019754 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.019718    2564 generic.go:356] "Generic (PLEG): container finished" podID="f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e" containerID="22f462ddbdb766e9a5aa6514623666aaa667cbdf0c5a1d9df691707288cac131" exitCode=0
Jul 13 15:38:45.019885 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.019773    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"
Jul 13 15:38:45.019885 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.019783    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t" event={"ID":"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e","Type":"ContainerDied","Data":"22f462ddbdb766e9a5aa6514623666aaa667cbdf0c5a1d9df691707288cac131"}
Jul 13 15:38:45.019885 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.019820    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t" event={"ID":"f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e","Type":"ContainerDied","Data":"2fe9a1117cbb0ae883198083b019260ec496ea27e03cb9c599cad1110298ca1c"}
Jul 13 15:38:45.019885 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.019838    2564 scope.go:117] "RemoveContainer" containerID="22f462ddbdb766e9a5aa6514623666aaa667cbdf0c5a1d9df691707288cac131"
Jul 13 15:38:45.028959 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.028941    2564 scope.go:117] "RemoveContainer" containerID="22f462ddbdb766e9a5aa6514623666aaa667cbdf0c5a1d9df691707288cac131"
Jul 13 15:38:45.029228 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:38:45.029210    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f462ddbdb766e9a5aa6514623666aaa667cbdf0c5a1d9df691707288cac131\": container with ID starting with 22f462ddbdb766e9a5aa6514623666aaa667cbdf0c5a1d9df691707288cac131 not found: ID does not exist" containerID="22f462ddbdb766e9a5aa6514623666aaa667cbdf0c5a1d9df691707288cac131"
Jul 13 15:38:45.029288 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.029238    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f462ddbdb766e9a5aa6514623666aaa667cbdf0c5a1d9df691707288cac131"} err="failed to get container status \"22f462ddbdb766e9a5aa6514623666aaa667cbdf0c5a1d9df691707288cac131\": rpc error: code = NotFound desc = could not find container \"22f462ddbdb766e9a5aa6514623666aaa667cbdf0c5a1d9df691707288cac131\": container with ID starting with 22f462ddbdb766e9a5aa6514623666aaa667cbdf0c5a1d9df691707288cac131 not found: ID does not exist"
Jul 13 15:38:45.090585 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.090553    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"]
Jul 13 15:38:45.096602 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.096568    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-88b181/rate-test-model-88b181-kserve-7c59c4844c-6zj4t"]
Jul 13 15:38:45.272364 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.272305    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:38:45.284723 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.284678    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-home\") pod \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") "
Jul 13 15:38:45.284863 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.284771    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-tmp-dir\") pod \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") "
Jul 13 15:38:45.284863 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.284813    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4gpg\" (UniqueName: \"kubernetes.io/projected/02576314-bdc4-43c9-8b5c-1e13cd9344b6-kube-api-access-v4gpg\") pod \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") "
Jul 13 15:38:45.284863 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.284840    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02576314-bdc4-43c9-8b5c-1e13cd9344b6-tls-certs\") pod \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") "
Jul 13 15:38:45.285034 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.284899    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-dshm\") pod \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") "
Jul 13 15:38:45.285034 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.284918    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-home" (OuterVolumeSpecName: "home") pod "02576314-bdc4-43c9-8b5c-1e13cd9344b6" (UID: "02576314-bdc4-43c9-8b5c-1e13cd9344b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:38:45.285034 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.284931    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-model-cache\") pod \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\" (UID: \"02576314-bdc4-43c9-8b5c-1e13cd9344b6\") "
Jul 13 15:38:45.285179 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.285064    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "02576314-bdc4-43c9-8b5c-1e13cd9344b6" (UID: "02576314-bdc4-43c9-8b5c-1e13cd9344b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:38:45.285179 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.285089    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "02576314-bdc4-43c9-8b5c-1e13cd9344b6" (UID: "02576314-bdc4-43c9-8b5c-1e13cd9344b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:38:45.285293 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.285274    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:38:45.285348 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.285295    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:38:45.285348 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.285309    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:38:45.287157 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.287129    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-dshm" (OuterVolumeSpecName: "dshm") pod "02576314-bdc4-43c9-8b5c-1e13cd9344b6" (UID: "02576314-bdc4-43c9-8b5c-1e13cd9344b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 15:38:45.287322 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.287305    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02576314-bdc4-43c9-8b5c-1e13cd9344b6-kube-api-access-v4gpg" (OuterVolumeSpecName: "kube-api-access-v4gpg") pod "02576314-bdc4-43c9-8b5c-1e13cd9344b6" (UID: "02576314-bdc4-43c9-8b5c-1e13cd9344b6"). InnerVolumeSpecName "kube-api-access-v4gpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:38:45.287394 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.287373    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02576314-bdc4-43c9-8b5c-1e13cd9344b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02576314-bdc4-43c9-8b5c-1e13cd9344b6" (UID: "02576314-bdc4-43c9-8b5c-1e13cd9344b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:38:45.386370 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.386327    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4gpg\" (UniqueName: \"kubernetes.io/projected/02576314-bdc4-43c9-8b5c-1e13cd9344b6-kube-api-access-v4gpg\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:38:45.386370 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.386364    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02576314-bdc4-43c9-8b5c-1e13cd9344b6-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:38:45.386370 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:45.386377    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02576314-bdc4-43c9-8b5c-1e13cd9344b6-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:38:46.026516 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.026478    2564 generic.go:356] "Generic (PLEG): container finished" podID="02576314-bdc4-43c9-8b5c-1e13cd9344b6" containerID="121ca3f3f760b90f405a88fb80762386b5671df973d5ab13486e1f417222e89c" exitCode=0
Jul 13 15:38:46.026721 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.026523    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"
Jul 13 15:38:46.026721 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.026542    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r" event={"ID":"02576314-bdc4-43c9-8b5c-1e13cd9344b6","Type":"ContainerDied","Data":"121ca3f3f760b90f405a88fb80762386b5671df973d5ab13486e1f417222e89c"}
Jul 13 15:38:46.026721 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.026581    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r" event={"ID":"02576314-bdc4-43c9-8b5c-1e13cd9344b6","Type":"ContainerDied","Data":"a2c3e4817da11d577651c5985d346865e79fbbe7947a5d79c9bef15b4b25ffb5"}
Jul 13 15:38:46.026721 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.026601    2564 scope.go:117] "RemoveContainer" containerID="121ca3f3f760b90f405a88fb80762386b5671df973d5ab13486e1f417222e89c"
Jul 13 15:38:46.036138 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.036123    2564 scope.go:117] "RemoveContainer" containerID="121ca3f3f760b90f405a88fb80762386b5671df973d5ab13486e1f417222e89c"
Jul 13 15:38:46.036382 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:38:46.036357    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121ca3f3f760b90f405a88fb80762386b5671df973d5ab13486e1f417222e89c\": container with ID starting with 121ca3f3f760b90f405a88fb80762386b5671df973d5ab13486e1f417222e89c not found: ID does not exist" containerID="121ca3f3f760b90f405a88fb80762386b5671df973d5ab13486e1f417222e89c"
Jul 13 15:38:46.036480 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.036387    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121ca3f3f760b90f405a88fb80762386b5671df973d5ab13486e1f417222e89c"} err="failed to get container status \"121ca3f3f760b90f405a88fb80762386b5671df973d5ab13486e1f417222e89c\": rpc error: code = NotFound desc = could not find container \"121ca3f3f760b90f405a88fb80762386b5671df973d5ab13486e1f417222e89c\": container with ID starting with 121ca3f3f760b90f405a88fb80762386b5671df973d5ab13486e1f417222e89c not found: ID does not exist"
Jul 13 15:38:46.048164 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.048140    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"]
Jul 13 15:38:46.054480 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.054459    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-450dce/rate-test-model-450dce-kserve-8564467cbc-6mx6r"]
Jul 13 15:38:46.316309 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.316235    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02576314-bdc4-43c9-8b5c-1e13cd9344b6" path="/var/lib/kubelet/pods/02576314-bdc4-43c9-8b5c-1e13cd9344b6/volumes"
Jul 13 15:38:46.316652 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.316638    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e" path="/var/lib/kubelet/pods/f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e/volumes"
Jul 13 15:38:46.900855 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.900824    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7715-140682982229312"]
Jul 13 15:38:46.901191 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.901180    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e" containerName="main"
Jul 13 15:38:46.901254 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.901193    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e" containerName="main"
Jul 13 15:38:46.901254 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.901204    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02576314-bdc4-43c9-8b5c-1e13cd9344b6" containerName="main"
Jul 13 15:38:46.901254 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.901211    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02576314-bdc4-43c9-8b5c-1e13cd9344b6" containerName="main"
Jul 13 15:38:46.901358 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.901285    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="02576314-bdc4-43c9-8b5c-1e13cd9344b6" containerName="main"
Jul 13 15:38:46.901358 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.901295    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f322cdb7-4ae0-4cfc-b7f8-9c8d1868d96e" containerName="main"
Jul 13 15:38:46.905579 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.905562    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682982229312"
Jul 13 15:38:46.908039 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.908021    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-27xqh\""
Jul 13 15:38:46.910576 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:46.910556    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7715-140682982229312"]
Jul 13 15:38:47.000133 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:47.000096    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqfmx\" (UniqueName: \"kubernetes.io/projected/fd87640b-75b2-4bea-aac1-554c01bbfa4f-kube-api-access-lqfmx\") pod \"test-curl-7715-140682982229312\" (UID: \"fd87640b-75b2-4bea-aac1-554c01bbfa4f\") " pod="openshift-ingress/test-curl-7715-140682982229312"
Jul 13 15:38:47.101571 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:47.101535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqfmx\" (UniqueName: \"kubernetes.io/projected/fd87640b-75b2-4bea-aac1-554c01bbfa4f-kube-api-access-lqfmx\") pod \"test-curl-7715-140682982229312\" (UID: \"fd87640b-75b2-4bea-aac1-554c01bbfa4f\") " pod="openshift-ingress/test-curl-7715-140682982229312"
Jul 13 15:38:47.110379 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:47.110351    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqfmx\" (UniqueName: \"kubernetes.io/projected/fd87640b-75b2-4bea-aac1-554c01bbfa4f-kube-api-access-lqfmx\") pod \"test-curl-7715-140682982229312\" (UID: \"fd87640b-75b2-4bea-aac1-554c01bbfa4f\") " pod="openshift-ingress/test-curl-7715-140682982229312"
Jul 13 15:38:47.215323 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:47.215244    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682982229312"
Jul 13 15:38:47.346282 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:47.346252    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7715-140682982229312"]
Jul 13 15:38:47.353086 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:38:47.352950    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd87640b_75b2_4bea_aac1_554c01bbfa4f.slice/crio-63ea85e6f750dd322a55c6298805cf5aeb14380428ee1ca7fbeebabcf2fbc097 WatchSource:0}: Error finding container 63ea85e6f750dd322a55c6298805cf5aeb14380428ee1ca7fbeebabcf2fbc097: Status 404 returned error can't find the container with id 63ea85e6f750dd322a55c6298805cf5aeb14380428ee1ca7fbeebabcf2fbc097
Jul 13 15:38:47.354145 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:47.354125    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 15:38:48.037858 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:48.037825    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7715-140682982229312" event={"ID":"fd87640b-75b2-4bea-aac1-554c01bbfa4f","Type":"ContainerStarted","Data":"63ea85e6f750dd322a55c6298805cf5aeb14380428ee1ca7fbeebabcf2fbc097"}
Jul 13 15:38:49.044319 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:49.044280    2564 generic.go:356] "Generic (PLEG): container finished" podID="fd87640b-75b2-4bea-aac1-554c01bbfa4f" containerID="221c922eb058c72a2c791e219c4cca4a7cd598350774cd556f91b84dd9266100" exitCode=0
Jul 13 15:38:49.044764 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:49.044369    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7715-140682982229312" event={"ID":"fd87640b-75b2-4bea-aac1-554c01bbfa4f","Type":"ContainerDied","Data":"221c922eb058c72a2c791e219c4cca4a7cd598350774cd556f91b84dd9266100"}
Jul 13 15:38:50.185860 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:50.185837    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682982229312"
Jul 13 15:38:50.224621 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:50.224587    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7715-140682982229312_fd87640b-75b2-4bea-aac1-554c01bbfa4f/test-curl-7715-140682982229312/0.log"
Jul 13 15:38:50.227827 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:50.227808    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqfmx\" (UniqueName: \"kubernetes.io/projected/fd87640b-75b2-4bea-aac1-554c01bbfa4f-kube-api-access-lqfmx\") pod \"fd87640b-75b2-4bea-aac1-554c01bbfa4f\" (UID: \"fd87640b-75b2-4bea-aac1-554c01bbfa4f\") "
Jul 13 15:38:50.230024 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:50.230002    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd87640b-75b2-4bea-aac1-554c01bbfa4f-kube-api-access-lqfmx" (OuterVolumeSpecName: "kube-api-access-lqfmx") pod "fd87640b-75b2-4bea-aac1-554c01bbfa4f" (UID: "fd87640b-75b2-4bea-aac1-554c01bbfa4f"). InnerVolumeSpecName "kube-api-access-lqfmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:38:50.250431 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:50.250404    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7715-140682982229312"]
Jul 13 15:38:50.254129 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:50.254109    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7715-140682982229312"]
Jul 13 15:38:50.316856 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:50.316792    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd87640b-75b2-4bea-aac1-554c01bbfa4f" path="/var/lib/kubelet/pods/fd87640b-75b2-4bea-aac1-554c01bbfa4f/volumes"
Jul 13 15:38:50.328574 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:50.328553    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqfmx\" (UniqueName: \"kubernetes.io/projected/fd87640b-75b2-4bea-aac1-554c01bbfa4f-kube-api-access-lqfmx\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:38:51.055617 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:51.055586    2564 scope.go:117] "RemoveContainer" containerID="221c922eb058c72a2c791e219c4cca4a7cd598350774cd556f91b84dd9266100"
Jul 13 15:38:51.055617 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:38:51.055615    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7715-140682982229312"
Jul 13 15:39:00.864784 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:00.864749    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"]
Jul 13 15:39:00.865315 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:00.864952    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24" podUID="6c2725fc-71ce-42c4-a633-a6bd1f16f58f" containerName="maas-api" containerID="cri-o://dc1fc8cee56e032c27495c5153e7b3b91caaae0bcd3f329276cbb3e03b9def44" gracePeriod=30
Jul 13 15:39:01.108213 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.108187    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:39:01.108741 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.108717    2564 generic.go:356] "Generic (PLEG): container finished" podID="6c2725fc-71ce-42c4-a633-a6bd1f16f58f" containerID="dc1fc8cee56e032c27495c5153e7b3b91caaae0bcd3f329276cbb3e03b9def44" exitCode=0
Jul 13 15:39:01.108851 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.108787    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24" event={"ID":"6c2725fc-71ce-42c4-a633-a6bd1f16f58f","Type":"ContainerDied","Data":"dc1fc8cee56e032c27495c5153e7b3b91caaae0bcd3f329276cbb3e03b9def44"}
Jul 13 15:39:01.108851 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.108842    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24" event={"ID":"6c2725fc-71ce-42c4-a633-a6bd1f16f58f","Type":"ContainerDied","Data":"422f51b2b214c4f83b9677f0bd391d7819d8dfb84b54a641c9de235454038ab1"}
Jul 13 15:39:01.108923 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.108863    2564 scope.go:117] "RemoveContainer" containerID="dc1fc8cee56e032c27495c5153e7b3b91caaae0bcd3f329276cbb3e03b9def44"
Jul 13 15:39:01.116879 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.116862    2564 scope.go:117] "RemoveContainer" containerID="dc1fc8cee56e032c27495c5153e7b3b91caaae0bcd3f329276cbb3e03b9def44"
Jul 13 15:39:01.117125 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:39:01.117107    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1fc8cee56e032c27495c5153e7b3b91caaae0bcd3f329276cbb3e03b9def44\": container with ID starting with dc1fc8cee56e032c27495c5153e7b3b91caaae0bcd3f329276cbb3e03b9def44 not found: ID does not exist" containerID="dc1fc8cee56e032c27495c5153e7b3b91caaae0bcd3f329276cbb3e03b9def44"
Jul 13 15:39:01.117176 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.117133    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1fc8cee56e032c27495c5153e7b3b91caaae0bcd3f329276cbb3e03b9def44"} err="failed to get container status \"dc1fc8cee56e032c27495c5153e7b3b91caaae0bcd3f329276cbb3e03b9def44\": rpc error: code = NotFound desc = could not find container \"dc1fc8cee56e032c27495c5153e7b3b91caaae0bcd3f329276cbb3e03b9def44\": container with ID starting with dc1fc8cee56e032c27495c5153e7b3b91caaae0bcd3f329276cbb3e03b9def44 not found: ID does not exist"
Jul 13 15:39:01.236108 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.236077    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqkj8\" (UniqueName: \"kubernetes.io/projected/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-kube-api-access-kqkj8\") pod \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\" (UID: \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\") "
Jul 13 15:39:01.236287 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.236162    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-openshift-service-ca\") pod \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\" (UID: \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\") "
Jul 13 15:39:01.236287 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.236198    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-maas-api-tls\") pod \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\" (UID: \"6c2725fc-71ce-42c4-a633-a6bd1f16f58f\") "
Jul 13 15:39:01.236502 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.236479    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c2725fc-71ce-42c4-a633-a6bd1f16f58f" (UID: "6c2725fc-71ce-42c4-a633-a6bd1f16f58f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:39:01.238283 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.238253    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-kube-api-access-kqkj8" (OuterVolumeSpecName: "kube-api-access-kqkj8") pod "6c2725fc-71ce-42c4-a633-a6bd1f16f58f" (UID: "6c2725fc-71ce-42c4-a633-a6bd1f16f58f"). InnerVolumeSpecName "kube-api-access-kqkj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:39:01.238416 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.238396    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c2725fc-71ce-42c4-a633-a6bd1f16f58f" (UID: "6c2725fc-71ce-42c4-a633-a6bd1f16f58f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:39:01.337279 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.337248    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:39:01.337279 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.337276    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:39:01.337279 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:01.337287    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqkj8\" (UniqueName: \"kubernetes.io/projected/6c2725fc-71ce-42c4-a633-a6bd1f16f58f-kube-api-access-kqkj8\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:39:02.114144 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:02.114113    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"
Jul 13 15:39:02.136617 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:02.136590    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"]
Jul 13 15:39:02.142456 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:02.142436    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-88b181-965b9ff87-4xs24"]
Jul 13 15:39:02.316762 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:02.316731    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c2725fc-71ce-42c4-a633-a6bd1f16f58f" path="/var/lib/kubelet/pods/6c2725fc-71ce-42c4-a633-a6bd1f16f58f/volumes"
Jul 13 15:39:16.740422 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:16.740337    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p"]
Jul 13 15:39:16.740929 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:16.740910    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c2725fc-71ce-42c4-a633-a6bd1f16f58f" containerName="maas-api"
Jul 13 15:39:16.741004 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:16.740933    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c2725fc-71ce-42c4-a633-a6bd1f16f58f" containerName="maas-api"
Jul 13 15:39:16.741004 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:16.740969    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd87640b-75b2-4bea-aac1-554c01bbfa4f" containerName="test-curl-7715-140682982229312"
Jul 13 15:39:16.741004 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:16.740979    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd87640b-75b2-4bea-aac1-554c01bbfa4f" containerName="test-curl-7715-140682982229312"
Jul 13 15:39:16.741161 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:16.741080    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd87640b-75b2-4bea-aac1-554c01bbfa4f" containerName="test-curl-7715-140682982229312"
Jul 13 15:39:16.741161 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:16.741094    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c2725fc-71ce-42c4-a633-a6bd1f16f58f" containerName="maas-api"
Jul 13 15:39:16.744491 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:16.744469    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p"
Jul 13 15:39:16.747093 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:16.747075    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hjd69\""
Jul 13 15:39:16.753234 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:16.753205    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p"]
Jul 13 15:39:16.872135 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:16.872097    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5480cb60-0609-4a25-aa23-8d8b3854b100-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p\" (UID: \"5480cb60-0609-4a25-aa23-8d8b3854b100\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p"
Jul 13 15:39:16.972781 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:16.972743    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5480cb60-0609-4a25-aa23-8d8b3854b100-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p\" (UID: \"5480cb60-0609-4a25-aa23-8d8b3854b100\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p"
Jul 13 15:39:16.973383 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:16.973361    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5480cb60-0609-4a25-aa23-8d8b3854b100-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p\" (UID: \"5480cb60-0609-4a25-aa23-8d8b3854b100\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p"
Jul 13 15:39:17.056174 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:17.056089    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p"
Jul 13 15:39:17.193125 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:17.193104    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p"]
Jul 13 15:39:17.195204 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:39:17.195173    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5480cb60_0609_4a25_aa23_8d8b3854b100.slice/crio-b65ffc581150dd19301ec9dd3d5c9eaceafbf415bf8377f439753999100b5751 WatchSource:0}: Error finding container b65ffc581150dd19301ec9dd3d5c9eaceafbf415bf8377f439753999100b5751: Status 404 returned error can't find the container with id b65ffc581150dd19301ec9dd3d5c9eaceafbf415bf8377f439753999100b5751
Jul 13 15:39:18.192293 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:18.192253    2564 generic.go:356] "Generic (PLEG): container finished" podID="5480cb60-0609-4a25-aa23-8d8b3854b100" containerID="5963657a111962afbcd45268a9bbaee8442baa30cb5c59642b18928040e36efa" exitCode=0
Jul 13 15:39:18.192757 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:18.192306    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p" event={"ID":"5480cb60-0609-4a25-aa23-8d8b3854b100","Type":"ContainerDied","Data":"5963657a111962afbcd45268a9bbaee8442baa30cb5c59642b18928040e36efa"}
Jul 13 15:39:18.192757 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:18.192369    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p" event={"ID":"5480cb60-0609-4a25-aa23-8d8b3854b100","Type":"ContainerStarted","Data":"b65ffc581150dd19301ec9dd3d5c9eaceafbf415bf8377f439753999100b5751"}
Jul 13 15:39:19.325964 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:19.325940    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p"
Jul 13 15:39:19.493952 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:19.493873    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5480cb60-0609-4a25-aa23-8d8b3854b100-maas-api-service-ca\") pod \"5480cb60-0609-4a25-aa23-8d8b3854b100\" (UID: \"5480cb60-0609-4a25-aa23-8d8b3854b100\") "
Jul 13 15:39:19.494216 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:19.494192    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5480cb60-0609-4a25-aa23-8d8b3854b100-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5480cb60-0609-4a25-aa23-8d8b3854b100" (UID: "5480cb60-0609-4a25-aa23-8d8b3854b100"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:39:19.595377 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:19.595342    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5480cb60-0609-4a25-aa23-8d8b3854b100-maas-api-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:39:20.203524 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:20.203485    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p" event={"ID":"5480cb60-0609-4a25-aa23-8d8b3854b100","Type":"ContainerDied","Data":"b65ffc581150dd19301ec9dd3d5c9eaceafbf415bf8377f439753999100b5751"}
Jul 13 15:39:20.203524 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:20.203525    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p"
Jul 13 15:39:20.203524 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:20.203533    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b65ffc581150dd19301ec9dd3d5c9eaceafbf415bf8377f439753999100b5751"
Jul 13 15:39:27.186028 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:27.185992    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"]
Jul 13 15:39:27.186442 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:27.186189    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t" podUID="083b6cd3-27c6-4d72-b052-9acbb035d16a" containerName="maas-api" containerID="cri-o://8575178a35fdd615c34d8c0bc8b9395554a246ec8adbb90982993b6ade45f888" gracePeriod=30
Jul 13 15:39:27.438748 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:27.438671    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:39:27.572072 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:27.572034    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083b6cd3-27c6-4d72-b052-9acbb035d16a-maas-api-tls\") pod \"083b6cd3-27c6-4d72-b052-9acbb035d16a\" (UID: \"083b6cd3-27c6-4d72-b052-9acbb035d16a\") "
Jul 13 15:39:27.572257 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:27.572088    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p96ll\" (UniqueName: \"kubernetes.io/projected/083b6cd3-27c6-4d72-b052-9acbb035d16a-kube-api-access-p96ll\") pod \"083b6cd3-27c6-4d72-b052-9acbb035d16a\" (UID: \"083b6cd3-27c6-4d72-b052-9acbb035d16a\") "
Jul 13 15:39:27.572257 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:27.572132    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083b6cd3-27c6-4d72-b052-9acbb035d16a-openshift-service-ca\") pod \"083b6cd3-27c6-4d72-b052-9acbb035d16a\" (UID: \"083b6cd3-27c6-4d72-b052-9acbb035d16a\") "
Jul 13 15:39:27.572548 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:27.572524    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083b6cd3-27c6-4d72-b052-9acbb035d16a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "083b6cd3-27c6-4d72-b052-9acbb035d16a" (UID: "083b6cd3-27c6-4d72-b052-9acbb035d16a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 15:39:27.574326 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:27.574301    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083b6cd3-27c6-4d72-b052-9acbb035d16a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "083b6cd3-27c6-4d72-b052-9acbb035d16a" (UID: "083b6cd3-27c6-4d72-b052-9acbb035d16a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 15:39:27.574393 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:27.574340    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083b6cd3-27c6-4d72-b052-9acbb035d16a-kube-api-access-p96ll" (OuterVolumeSpecName: "kube-api-access-p96ll") pod "083b6cd3-27c6-4d72-b052-9acbb035d16a" (UID: "083b6cd3-27c6-4d72-b052-9acbb035d16a"). InnerVolumeSpecName "kube-api-access-p96ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 15:39:27.673626 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:27.673584    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p96ll\" (UniqueName: \"kubernetes.io/projected/083b6cd3-27c6-4d72-b052-9acbb035d16a-kube-api-access-p96ll\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:39:27.673626 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:27.673617    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083b6cd3-27c6-4d72-b052-9acbb035d16a-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:39:27.673626 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:27.673629    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083b6cd3-27c6-4d72-b052-9acbb035d16a-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Jul 13 15:39:28.245162 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:28.245124    2564 generic.go:356] "Generic (PLEG): container finished" podID="083b6cd3-27c6-4d72-b052-9acbb035d16a" containerID="8575178a35fdd615c34d8c0bc8b9395554a246ec8adbb90982993b6ade45f888" exitCode=0
Jul 13 15:39:28.245545 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:28.245170    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"
Jul 13 15:39:28.245545 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:28.245219    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t" event={"ID":"083b6cd3-27c6-4d72-b052-9acbb035d16a","Type":"ContainerDied","Data":"8575178a35fdd615c34d8c0bc8b9395554a246ec8adbb90982993b6ade45f888"}
Jul 13 15:39:28.245545 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:28.245278    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t" event={"ID":"083b6cd3-27c6-4d72-b052-9acbb035d16a","Type":"ContainerDied","Data":"c6ddbb981e0f0e5d1fb8567b2a93d299c7e544a46e30b3acf6d50eb56a47d189"}
Jul 13 15:39:28.245545 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:28.245305    2564 scope.go:117] "RemoveContainer" containerID="8575178a35fdd615c34d8c0bc8b9395554a246ec8adbb90982993b6ade45f888"
Jul 13 15:39:28.254278 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:28.254259    2564 scope.go:117] "RemoveContainer" containerID="8575178a35fdd615c34d8c0bc8b9395554a246ec8adbb90982993b6ade45f888"
Jul 13 15:39:28.254563 ip-10-0-128-29 kubenswrapper[2564]: E0713 15:39:28.254545    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8575178a35fdd615c34d8c0bc8b9395554a246ec8adbb90982993b6ade45f888\": container with ID starting with 8575178a35fdd615c34d8c0bc8b9395554a246ec8adbb90982993b6ade45f888 not found: ID does not exist" containerID="8575178a35fdd615c34d8c0bc8b9395554a246ec8adbb90982993b6ade45f888"
Jul 13 15:39:28.254622 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:28.254573    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8575178a35fdd615c34d8c0bc8b9395554a246ec8adbb90982993b6ade45f888"} err="failed to get container status \"8575178a35fdd615c34d8c0bc8b9395554a246ec8adbb90982993b6ade45f888\": rpc error: code = NotFound desc = could not find container \"8575178a35fdd615c34d8c0bc8b9395554a246ec8adbb90982993b6ade45f888\": container with ID starting with 8575178a35fdd615c34d8c0bc8b9395554a246ec8adbb90982993b6ade45f888 not found: ID does not exist"
Jul 13 15:39:28.266913 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:28.266885    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"]
Jul 13 15:39:28.272424 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:28.272398    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-450dce-7b7cbdfdfc-q756t"]
Jul 13 15:39:28.318419 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:28.318385    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="083b6cd3-27c6-4d72-b052-9acbb035d16a" path="/var/lib/kubelet/pods/083b6cd3-27c6-4d72-b052-9acbb035d16a/volumes"
Jul 13 15:39:43.318760 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:43.318618    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c7db64446-4k2n7_59192a8a-42f2-45f5-b8a4-1bfe09292ef7/authorino/0.log"
Jul 13 15:39:48.738083 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:48.738055    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-khsgl_f4d9eaad-dda7-4a58-96e9-e7bafc50e442/manager/0.log"
Jul 13 15:39:49.637573 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:49.637543    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67985f7df8-pqjp8_1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f/copy-manifests/0.log"
Jul 13 15:39:49.645437 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:49.645408    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67985f7df8-pqjp8_1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f/manager/0.log"
Jul 13 15:39:50.420887 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:50.420863    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-76c770f3-wxd6r_42d816e2-bf78-49ad-a03d-7e0d8eda90f8/revoke-keys/0.log"
Jul 13 15:39:50.539985 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:50.539954    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-d0b8e9a7-xbwd7_67ff49c2-f7f8-4651-b7ce-c6a1b70c34aa/revoke-keys/1.log"
Jul 13 15:39:51.265168 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:51.265139    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-15019914-mrbs6_d29e0619-2a2c-44af-b276-8880be8db05e/revoke-keys/0.log"
Jul 13 15:39:51.381493 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:51.381457    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-f969cfbe-zzmhq_18d6e559-6482-4a46-b864-f13115e1f85c/revoke-keys/0.log"
Jul 13 15:39:51.743998 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:51.743964    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-450dce-bhr5p_5480cb60-0609-4a25-aa23-8d8b3854b100/revoke-keys/0.log"
Jul 13 15:39:53.269651 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:53.269623    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c7db64446-4k2n7_59192a8a-42f2-45f5-b8a4-1bfe09292ef7/authorino/0.log"
Jul 13 15:39:53.684251 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:53.684220    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lfwwx_caa479ca-9183-4c1f-8b5e-ed55fc4e8822/kuadrant-console-plugin/0.log"
Jul 13 15:39:54.221911 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:54.221878    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-fpwvv_49e17671-ccfe-4d04-8f3a-9d979030c676/manager/0.log"
Jul 13 15:39:54.862242 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:54.862159    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cfcb69fbd-mqqsn_eecea63b-388b-4f4c-8db4-368cedb166ac/kube-auth-proxy/0.log"
Jul 13 15:39:55.876454 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:55.876404    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-9jgc6_fbb2ffcb-5103-4e3e-b3e9-debacba0fc7b/main/0.log"
Jul 13 15:39:56.255654 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:39:56.255626    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzgk2p_c6765fa4-2372-4092-bc64-2601b628519a/main/0.log"
Jul 13 15:40:03.360922 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:03.360896    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jgpsr_bdef54cb-b0a3-4783-9655-5296a4f95022/global-pull-secret-syncer/0.log"
Jul 13 15:40:03.538412 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:03.538385    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tjrpn_4bd78a31-4667-420c-9ed6-52df573dbaf5/konnectivity-agent/0.log"
Jul 13 15:40:03.562843 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:03.562819    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-29.ec2.internal_9fe4ee3c51c80837e4836133e719eda0/haproxy/0.log"
Jul 13 15:40:08.574641 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:08.574605    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c7db64446-4k2n7_59192a8a-42f2-45f5-b8a4-1bfe09292ef7/authorino/0.log"
Jul 13 15:40:08.691227 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:08.691200    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lfwwx_caa479ca-9183-4c1f-8b5e-ed55fc4e8822/kuadrant-console-plugin/0.log"
Jul 13 15:40:09.079461 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:09.079378    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-fpwvv_49e17671-ccfe-4d04-8f3a-9d979030c676/manager/0.log"
Jul 13 15:40:10.848521 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:10.848489    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-jd8gw_5471e5ad-8965-4379-af6a-2cb21fb280e7/kube-state-metrics/0.log"
Jul 13 15:40:10.872369 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:10.872341    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-jd8gw_5471e5ad-8965-4379-af6a-2cb21fb280e7/kube-rbac-proxy-main/0.log"
Jul 13 15:40:10.908561 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:10.908538    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-jd8gw_5471e5ad-8965-4379-af6a-2cb21fb280e7/kube-rbac-proxy-self/0.log"
Jul 13 15:40:10.966896 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:10.966870    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-z7g2s_9b8aa987-baea-42b4-814c-c92754534afc/monitoring-plugin/0.log"
Jul 13 15:40:10.996039 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:10.995960    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2mnl5_579a5f61-d453-4327-ab8d-ea6dae5d44c9/node-exporter/0.log"
Jul 13 15:40:11.020230 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.020207    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2mnl5_579a5f61-d453-4327-ab8d-ea6dae5d44c9/kube-rbac-proxy/0.log"
Jul 13 15:40:11.045246 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.045207    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2mnl5_579a5f61-d453-4327-ab8d-ea6dae5d44c9/init-textfile/0.log"
Jul 13 15:40:11.250570 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.250493    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5w66h_e8f2fc05-e0b7-4131-b42b-41aa944c7679/kube-rbac-proxy-main/0.log"
Jul 13 15:40:11.291862 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.291836    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5w66h_e8f2fc05-e0b7-4131-b42b-41aa944c7679/kube-rbac-proxy-self/0.log"
Jul 13 15:40:11.329168 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.329142    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5w66h_e8f2fc05-e0b7-4131-b42b-41aa944c7679/openshift-state-metrics/0.log"
Jul 13 15:40:11.387516 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.387492    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bb33220c-5612-423e-85a6-3273a704a8bf/prometheus/0.log"
Jul 13 15:40:11.426069 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.426039    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bb33220c-5612-423e-85a6-3273a704a8bf/config-reloader/0.log"
Jul 13 15:40:11.461704 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.461673    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bb33220c-5612-423e-85a6-3273a704a8bf/thanos-sidecar/0.log"
Jul 13 15:40:11.500037 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.500007    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bb33220c-5612-423e-85a6-3273a704a8bf/kube-rbac-proxy-web/0.log"
Jul 13 15:40:11.541894 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.541823    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bb33220c-5612-423e-85a6-3273a704a8bf/kube-rbac-proxy/0.log"
Jul 13 15:40:11.581925 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.581904    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bb33220c-5612-423e-85a6-3273a704a8bf/kube-rbac-proxy-thanos/0.log"
Jul 13 15:40:11.633079 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.633053    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bb33220c-5612-423e-85a6-3273a704a8bf/init-config-reloader/0.log"
Jul 13 15:40:11.696375 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.696343    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-x96cn_32357f5b-ef2c-449c-902f-c4db8304ca83/prometheus-operator/0.log"
Jul 13 15:40:11.732840 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.732804    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-x96cn_32357f5b-ef2c-449c-902f-c4db8304ca83/kube-rbac-proxy/0.log"
Jul 13 15:40:11.957328 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.957288    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77bfd56947-67k5j_7d2ac378-5a0a-48d0-8ba1-950196dd7efc/thanos-query/0.log"
Jul 13 15:40:11.995963 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:11.995939    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77bfd56947-67k5j_7d2ac378-5a0a-48d0-8ba1-950196dd7efc/kube-rbac-proxy-web/0.log"
Jul 13 15:40:12.034414 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:12.034385    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77bfd56947-67k5j_7d2ac378-5a0a-48d0-8ba1-950196dd7efc/kube-rbac-proxy/0.log"
Jul 13 15:40:12.074524 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:12.074497    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77bfd56947-67k5j_7d2ac378-5a0a-48d0-8ba1-950196dd7efc/prom-label-proxy/0.log"
Jul 13 15:40:12.114896 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:12.114865    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77bfd56947-67k5j_7d2ac378-5a0a-48d0-8ba1-950196dd7efc/kube-rbac-proxy-rules/0.log"
Jul 13 15:40:12.154830 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:12.154799    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77bfd56947-67k5j_7d2ac378-5a0a-48d0-8ba1-950196dd7efc/kube-rbac-proxy-metrics/0.log"
Jul 13 15:40:14.012251 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.012218    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"]
Jul 13 15:40:14.012779 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.012568    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5480cb60-0609-4a25-aa23-8d8b3854b100" containerName="revoke-keys"
Jul 13 15:40:14.012779 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.012579    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5480cb60-0609-4a25-aa23-8d8b3854b100" containerName="revoke-keys"
Jul 13 15:40:14.012779 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.012589    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083b6cd3-27c6-4d72-b052-9acbb035d16a" containerName="maas-api"
Jul 13 15:40:14.012779 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.012594    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="083b6cd3-27c6-4d72-b052-9acbb035d16a" containerName="maas-api"
Jul 13 15:40:14.012779 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.012650    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="083b6cd3-27c6-4d72-b052-9acbb035d16a" containerName="maas-api"
Jul 13 15:40:14.012779 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.012661    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5480cb60-0609-4a25-aa23-8d8b3854b100" containerName="revoke-keys"
Jul 13 15:40:14.015817 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.015797    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.019145 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.019125    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vdpg8\"/\"openshift-service-ca.crt\""
Jul 13 15:40:14.019955 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.019940    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vdpg8\"/\"default-dockercfg-sdvkg\""
Jul 13 15:40:14.020025 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.019979    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vdpg8\"/\"kube-root-ca.crt\""
Jul 13 15:40:14.026871 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.026852    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"]
Jul 13 15:40:14.087402 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.087371    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-sys\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.087581 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.087409    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56lk7\" (UniqueName: \"kubernetes.io/projected/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-kube-api-access-56lk7\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.087581 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.087451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-podres\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.087581 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.087539    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-lib-modules\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.087781 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.087603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-proc\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.188468 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.188438    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-proc\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.188468 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.188472    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-sys\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.188734 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.188491    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56lk7\" (UniqueName: \"kubernetes.io/projected/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-kube-api-access-56lk7\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.188734 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.188536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-podres\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.188734 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.188572    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-proc\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.188734 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.188581    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-sys\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.188734 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.188588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-lib-modules\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.188734 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.188719    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-lib-modules\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.188972 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.188760    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-podres\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.197161 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.197137    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56lk7\" (UniqueName: \"kubernetes.io/projected/ff98a522-22a2-4dbf-ac3e-c4e7b4016a02-kube-api-access-56lk7\") pod \"perf-node-gather-daemonset-dx74t\" (UID: \"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02\") " pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.325448 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.325372    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:14.463637 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:14.463595    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"]
Jul 13 15:40:14.466871 ip-10-0-128-29 kubenswrapper[2564]: W0713 15:40:14.466840    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podff98a522_22a2_4dbf_ac3e_c4e7b4016a02.slice/crio-e4029761931f65fc59940e4622022dea6abd8b6dce60ce788e008c9abd6fcb05 WatchSource:0}: Error finding container e4029761931f65fc59940e4622022dea6abd8b6dce60ce788e008c9abd6fcb05: Status 404 returned error can't find the container with id e4029761931f65fc59940e4622022dea6abd8b6dce60ce788e008c9abd6fcb05
Jul 13 15:40:15.476014 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:15.475977    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t" event={"ID":"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02","Type":"ContainerStarted","Data":"fece1f8090a9caa419807be31475d6e0874b33e4dd4e3d7af481bef68c378f88"}
Jul 13 15:40:15.476399 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:15.476034    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t" event={"ID":"ff98a522-22a2-4dbf-ac3e-c4e7b4016a02","Type":"ContainerStarted","Data":"e4029761931f65fc59940e4622022dea6abd8b6dce60ce788e008c9abd6fcb05"}
Jul 13 15:40:15.476399 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:15.476076    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:15.493629 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:15.493585    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t" podStartSLOduration=2.493570749 podStartE2EDuration="2.493570749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 15:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:40:15.492734957 +0000 UTC m=+2841.689887904" watchObservedRunningTime="2026-07-13 15:40:15.493570749 +0000 UTC m=+2841.690723696"
Jul 13 15:40:16.161232 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:16.161197    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4dd6p_33636c73-08ef-4a53-9365-76554fce2144/dns/0.log"
Jul 13 15:40:16.185488 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:16.185460    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4dd6p_33636c73-08ef-4a53-9365-76554fce2144/kube-rbac-proxy/0.log"
Jul 13 15:40:16.367110 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:16.367080    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ptz2c_84f22329-522f-4502-a166-d0f7139138fc/dns-node-resolver/0.log"
Jul 13 15:40:16.990113 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:16.990085    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k9gzt_7ef9f15a-5b1d-46d5-8e1c-053c4cc891a5/node-ca/0.log"
Jul 13 15:40:18.231159 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:18.231131    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cfcb69fbd-mqqsn_eecea63b-388b-4f4c-8db4-368cedb166ac/kube-auth-proxy/0.log"
Jul 13 15:40:19.177111 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:19.177084    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5xl9h_7b085a1b-e5a1-4106-a6a3-e7ada1a20f82/serve-healthcheck-canary/0.log"
Jul 13 15:40:19.837179 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:19.837150    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nf6sd_c680c3f4-6691-4a7b-ac7b-8fdf893a832d/kube-rbac-proxy/0.log"
Jul 13 15:40:19.859736 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:19.859689    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nf6sd_c680c3f4-6691-4a7b-ac7b-8fdf893a832d/exporter/0.log"
Jul 13 15:40:19.882863 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:19.882838    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nf6sd_c680c3f4-6691-4a7b-ac7b-8fdf893a832d/extractor/0.log"
Jul 13 15:40:21.491289 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:21.491262    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vdpg8/perf-node-gather-daemonset-dx74t"
Jul 13 15:40:22.078592 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:22.078557    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-khsgl_f4d9eaad-dda7-4a58-96e9-e7bafc50e442/manager/0.log"
Jul 13 15:40:22.738468 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:22.738444    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67985f7df8-pqjp8_1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f/manager/0.log"
Jul 13 15:40:22.763818 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:22.763794    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67985f7df8-pqjp8_1cbaabdc-af73-42ba-8d7f-d1b04a2f4f4f/copy-manifests/0.log"
Jul 13 15:40:23.953100 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:23.953068    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cfcb69fbd-mqqsn_eecea63b-388b-4f4c-8db4-368cedb166ac/kube-auth-proxy/0.log"
Jul 13 15:40:24.254556 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:24.254455    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-677c5d8dc-pw7xx_c944fa94-c547-492d-b187-b033865f6951/manager/0.log"
Jul 13 15:40:24.311933 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:24.311899    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-2m7bl_3f081c03-dad4-4d64-adcd-6c31d8a1d021/openshift-lws-operator/0.log"
Jul 13 15:40:25.492590 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:25.492564    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cfcb69fbd-mqqsn_eecea63b-388b-4f4c-8db4-368cedb166ac/kube-auth-proxy/0.log"
Jul 13 15:40:31.355763 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:31.355735    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wgw48_f6542e3b-cdda-4bd2-9419-667e66525978/kube-multus-additional-cni-plugins/0.log"
Jul 13 15:40:31.381459 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:31.381432    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wgw48_f6542e3b-cdda-4bd2-9419-667e66525978/egress-router-binary-copy/0.log"
Jul 13 15:40:31.406082 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:31.406054    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wgw48_f6542e3b-cdda-4bd2-9419-667e66525978/cni-plugins/0.log"
Jul 13 15:40:31.431863 ip-10-0-128-29 kubenswrapper[2564]: I0713 15:40:31.431837    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wgw48_f6542e3b-cdda-4bd2-9419-667e66525978/bond-cni-plugin/0.log"