Jul 14 14:30:22.205775 ip-10-0-141-209 systemd[1]: Starting Kubernetes Kubelet...
Jul 14 14:30:22.646315 ip-10-0-141-209 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 14:30:22.646315 ip-10-0-141-209 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 14 14:30:22.646315 ip-10-0-141-209 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 14:30:22.646315 ip-10-0-141-209 kubenswrapper[2586]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 14 14:30:22.646315 ip-10-0-141-209 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 14:30:22.649129 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.649045    2586 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 14 14:30:22.653984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.653968    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 14:30:22.653984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.653983    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.653988    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.653991    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.653993    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.653997    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654001    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654005    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654009    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654016    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654019    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654022    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654025    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654028    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654030    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654032    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654035    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654037    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654040    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654043    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 14:30:22.654048 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654045    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654048    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654050    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654053    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654056    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654058    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654061    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654064    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654066    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654070    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654074    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654077    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654080    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654082    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654085    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654087    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654090    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654092    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654094    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654096    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 14:30:22.654503 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654099    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654102    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654104    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654106    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654109    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654111    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654114    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654116    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654119    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654121    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654123    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654125    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654128    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654130    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654133    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654136    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654138    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654141    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654143    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654146    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 14:30:22.654965 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654148    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654151    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654153    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654155    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654158    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654160    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654163    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654165    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654167    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654170    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654172    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654175    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654177    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654179    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654182    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654184    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654187    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654189    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654192    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654195    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 14:30:22.655448 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654198    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654200    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654202    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654205    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654209    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654212    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654215    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654218    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654221    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654224    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654227    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654230    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654233    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654235    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654237    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654240    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654242    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654244    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654246    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 14:30:22.655913 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654977    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654984    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654986    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654989    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654993    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654995    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.654998    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655000    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655006    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655010    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655014    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655018    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655021    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655024    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655027    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655030    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655033    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655035    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655038    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 14:30:22.656355 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655042    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655045    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655048    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655050    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655053    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655055    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655058    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655061    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655063    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655065    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655068    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655070    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655073    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655075    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655077    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655080    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655082    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655085    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655087    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655090    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 14:30:22.656866 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655092    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655094    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655097    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655100    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655102    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655104    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655107    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655109    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655112    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655114    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655116    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655118    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655122    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655126    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655128    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655131    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655133    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655136    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655139    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655141    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 14:30:22.657337 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655143    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655146    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655148    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655151    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655153    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655155    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655158    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655160    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655162    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655165    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655168    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655171    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655173    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655175    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655178    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655181    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655184    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655187    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655189    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655192    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 14:30:22.657841 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655194    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655197    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655199    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655202    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655205    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655207    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655209    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655212    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655214    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655217    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655219    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655221    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655223    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655226    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655228    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655230    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655233    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655235    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655237    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 14:30:22.658315 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655241    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655614    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655620    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655623    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655625    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655628    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655631    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655633    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655636    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655639    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655642    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655645    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655648    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655650    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655653    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655656    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655658    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655666    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655669    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655672    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 14:30:22.658783 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655675    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655677    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655680    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655683    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655685    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655688    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655690    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655693    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655695    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655698    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655700    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655702    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655706    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655709    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655712    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655715    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655717    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655720    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655723    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 14:30:22.659258 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655725    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655728    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655731    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655734    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655739    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655742    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655745    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655748    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655750    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655753    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655757    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655761    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655763    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655765    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655768    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655770    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655773    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655775    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655777    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 14:30:22.659724 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655780    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655782    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655784    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655787    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655789    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655791    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655794    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655796    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655799    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655801    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655803    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655806    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655808    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655810    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655813    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655815    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655819    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655821    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655823    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 14:30:22.660176 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655826    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655828    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655830    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655833    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655837    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655839    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655842    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655844    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655847    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655849    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655852    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655854    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655856    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655859    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655861    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655864    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655866    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655868    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655871    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655873    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 14:30:22.660661 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655876    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655878    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.655880    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.655954    2586 flags.go:64] FLAG: --address="0.0.0.0"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.655961    2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.655967    2586 flags.go:64] FLAG: --anonymous-auth="true"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.655972    2586 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.655976    2586 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.655980    2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.655985    2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.655990    2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.655993    2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.655996    2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.655999    2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656003    2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656006    2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656009    2586 flags.go:64] FLAG: --cgroup-root=""
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656012    2586 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656015    2586 flags.go:64] FLAG: --client-ca-file=""
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656018    2586 flags.go:64] FLAG: --cloud-provider="external"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656021    2586 flags.go:64] FLAG: --cluster-dns="[]"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656024    2586 flags.go:64] FLAG: --cluster-domain=""
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656027    2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656030    2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 14 14:30:22.661122 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656033    2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656036    2586 flags.go:64] FLAG: --container-log-max-files="5"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656039    2586 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656042    2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656045    2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656049    2586 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656052    2586 flags.go:64] FLAG: --contention-profiling="false"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656054    2586 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656057    2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656060    2586 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656063    2586 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656067    2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656070    2586 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656072    2586 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656075    2586 flags.go:64] FLAG: --enable-load-reader="false"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656078    2586 flags.go:64] FLAG: --enable-server="true"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656081    2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656087    2586 flags.go:64] FLAG: --event-burst="100"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656090    2586 flags.go:64] FLAG: --event-qps="50"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656094    2586 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656097    2586 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656100    2586 flags.go:64] FLAG: --eviction-hard=""
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656104    2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656107    2586 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656110    2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 14 14:30:22.661738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656113    2586 flags.go:64] FLAG: --eviction-soft=""
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656116    2586 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656118    2586 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656121    2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656124    2586 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656127    2586 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656130    2586 flags.go:64] FLAG: --fail-swap-on="true"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656132    2586 flags.go:64] FLAG: --feature-gates=""
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656136    2586 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656139    2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656142    2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656144    2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656148    2586 flags.go:64] FLAG: --healthz-port="10248"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656150    2586 flags.go:64] FLAG: --help="false"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656153    2586 flags.go:64] FLAG: --hostname-override="ip-10-0-141-209.ec2.internal"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656156    2586 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656159    2586 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656176    2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656180    2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656184    2586 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656187    2586 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656190    2586 flags.go:64] FLAG: --image-service-endpoint=""
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656193    2586 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 14 14:30:22.662323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656196    2586 flags.go:64] FLAG: --kube-api-burst="100"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656200    2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656203    2586 flags.go:64] FLAG: --kube-api-qps="50"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656206    2586 flags.go:64] FLAG: --kube-reserved=""
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656209    2586 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656212    2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656214    2586 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656217    2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656220    2586 flags.go:64] FLAG: --lock-file=""
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656223    2586 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656226    2586 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656228    2586 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656233    2586 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656236    2586 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656239    2586 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656242    2586 flags.go:64] FLAG: --logging-format="text"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656244    2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656247    2586 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656250    2586 flags.go:64] FLAG: --manifest-url=""
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656253    2586 flags.go:64] FLAG: --manifest-url-header=""
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656257    2586 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656260    2586 flags.go:64] FLAG: --max-open-files="1000000"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656263    2586 flags.go:64] FLAG: --max-pods="110"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656266    2586 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656269    2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 14 14:30:22.662910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656272    2586 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656275    2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656278    2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656280    2586 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656283    2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656290    2586 flags.go:64] FLAG: --node-status-max-images="50"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656293    2586 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656296    2586 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656300    2586 flags.go:64] FLAG: --pod-cidr=""
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656303    2586 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656308    2586 flags.go:64] FLAG: --pod-manifest-path=""
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656311    2586 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656314    2586 flags.go:64] FLAG: --pods-per-core="0"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656317    2586 flags.go:64] FLAG: --port="10250"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656320    2586 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656323    2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00955843abc64b563"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656326    2586 flags.go:64] FLAG: --qos-reserved=""
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656329    2586 flags.go:64] FLAG: --read-only-port="10255"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656332    2586 flags.go:64] FLAG: --register-node="true"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656335    2586 flags.go:64] FLAG: --register-with-taints=""
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656338    2586 flags.go:64] FLAG: --registry-burst="10"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656341    2586 flags.go:64] FLAG: --registry-qps="5"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656344    2586 flags.go:64] FLAG: --reserved-cpus=""
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656347    2586 flags.go:64] FLAG: --reserved-memory=""
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656350    2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 14 14:30:22.663558 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656353    2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656356    2586 flags.go:64] FLAG: --rotate-certificates="false"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656359    2586 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656361    2586 flags.go:64] FLAG: --runonce="false"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656374    2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656377    2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656380    2586 flags.go:64] FLAG: --seccomp-default="false"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656384    2586 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656386    2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656389    2586 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656392    2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656395    2586 flags.go:64] FLAG: --storage-driver-password="root"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656398    2586 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656401    2586 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656404    2586 flags.go:64] FLAG: --storage-driver-user="root"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656406    2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656424    2586 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656428    2586 flags.go:64] FLAG: --system-cgroups=""
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656430    2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656435    2586 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656439    2586 flags.go:64] FLAG: --tls-cert-file=""
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656442    2586 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656446    2586 flags.go:64] FLAG: --tls-min-version=""
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656449    2586 flags.go:64] FLAG: --tls-private-key-file=""
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656452    2586 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 14 14:30:22.664437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656454    2586 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656457    2586 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656460    2586 flags.go:64] FLAG: --v="2"
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656466    2586 flags.go:64] FLAG: --version="false"
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656470    2586 flags.go:64] FLAG: --vmodule=""
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656473    2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.656476    2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656647    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656652    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656655    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656658    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656661    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656664    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656666    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656669    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656671    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656674    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656676    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656679    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656681    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656684    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 14:30:22.665430 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656686    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656688    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656691    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656693    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656696    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656698    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656701    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656704    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656708    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656711    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656713    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656716    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656718    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656721    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656723    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656725    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656728    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656730    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656732    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656735    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 14:30:22.666381 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656738    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656742    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656745    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656747    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656750    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656752    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656755    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656759    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656762    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656765    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656767    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656770    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656773    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656776    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656778    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656781    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656784    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656787    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656789    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 14:30:22.666984 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656791    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656795    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656798    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656800    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656803    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656805    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656807    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656810    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656812    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656815    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656817    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656820    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656822    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656824    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656826    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656829    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656831    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656833    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656836    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656838    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 14:30:22.667468 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656840    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656843    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656845    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656847    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656849    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656852    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656854    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656856    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656859    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656861    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656864    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656866    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656869    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656873    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656876    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656880    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656883    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656886    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656888    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656891    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 14:30:22.667951 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656893    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656896    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656898    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656900    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656903    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.656905    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.657596    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.667545    2586 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.667562    2586 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.667581    2586 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.667590    2586 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667650    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667655    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667659    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667663    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 14:30:22.668434 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667666    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667668    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667671    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667674    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667676    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667679    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667681    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667684    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667686    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667689    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667692    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667695    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667697    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667700    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667702    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667705    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667707    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667710    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667712    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667715    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 14:30:22.668819 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667718    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667720    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667723    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667725    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667728    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667731    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667734    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667737    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667739    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667743    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667747    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667750    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667752    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667755    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667757    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667760    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667763    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667767    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667770    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 14:30:22.669298 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667772    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667775    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667778    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667781    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667783    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667786    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667788    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667791    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667794    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667796    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667799    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667802    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667806    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667809    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667811    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667814    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667816    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667818    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667821    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 14:30:22.669784 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667824    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667826    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667829    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667831    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667834    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667837    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667840    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667842    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667845    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667847    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667849    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667852    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667855    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667857    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667859    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667862    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667865    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667868    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667870    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 14:30:22.670236 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667873    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667875    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667877    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667880    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667882    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667885    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667887    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667890    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667892    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667895    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667897    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667899    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667902    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667905    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667908    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667910    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667913    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 14:30:22.670705 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.667915    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.667920    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668040    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668046    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668048    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668051    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668054    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668057    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668060    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668062    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668065    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668067    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668070    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668072    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668077    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 14:30:22.671132 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668081    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668084    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668087    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668089    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668092    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668094    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668097    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668099    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668101    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668104    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668106    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668109    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668111    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668114    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668116    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668119    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668121    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668124    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668126    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 14:30:22.671547 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668129    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668133    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668136    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668139    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668142    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668144    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668146    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668149    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668151    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668154    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668157    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668159    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668162    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668165    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668167    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668170    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668172    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668174    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668177    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 14:30:22.672013 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668179    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668182    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668184    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668186    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668189    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668191    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668194    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668196    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668199    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668203    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668206    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668209    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668211    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668214    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668216    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668219    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668222    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668224    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668227    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668229    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 14:30:22.672474 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668231    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668234    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668236    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668241    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668243    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668246    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668248    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668251    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668254    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668256    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668259    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668261    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668263    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668266    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668268    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668271    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668273    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668276    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668278    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668280    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 14:30:22.672950 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668283    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 14:30:22.673445 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668285    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 14:30:22.673445 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668287    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 14:30:22.673445 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668290    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 14:30:22.673445 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668292    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 14:30:22.673445 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668295    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 14:30:22.673445 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668297    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 14:30:22.673445 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:22.668299    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 14:30:22.673445 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.668304    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 14:30:22.673445 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.668459    2586 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 14 14:30:22.673445 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.671025    2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 14 14:30:22.673445 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.672767    2586 server.go:1019] "Starting client certificate rotation"
Jul 14 14:30:22.673445 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.672865    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 14:30:22.673445 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.672904    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 14:30:22.675612 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.675598    2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 14:30:22.676863 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.676833    2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 14:30:22.693958 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.693937    2586 log.go:25] "Validated CRI v1 runtime API"
Jul 14 14:30:22.696296 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.696281    2586 log.go:25] "Validated CRI v1 image API"
Jul 14 14:30:22.697192 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.697178    2586 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 14 14:30:22.705297 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.705276    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 14:30:22.706983 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.706964    2586 fs.go:135] Filesystem UUIDs: map[05efe1b3-f175-42bc-ba42-b17c8a94da62:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d3196301-0879-446a-bde7-279da847aa10:/dev/nvme0n1p4]
Jul 14 14:30:22.707036 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.706983    2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 14 14:30:22.714491 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.714367    2586 manager.go:216] Machine: {Timestamp:2026-07-14 14:30:22.712910424 +0000 UTC m=+0.379096837 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098992 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29fbacbc5b004ae58e12dc21d71799 SystemUUID:ec29fbac-bc5b-004a-e58e-12dc21d71799 BootID:05db09db-d215-4eb3-9bf7-eab31282b5d1 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:45:e3:68:e3:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:45:e3:68:e3:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:79:dd:52:e9:3f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 14 14:30:22.714491 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.714479    2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 14 14:30:22.714612 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.714559    2586 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 14 14:30:22.716678 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.716653    2586 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 14 14:30:22.716816 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.716681    2586 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-209.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 14 14:30:22.718217 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.718206    2586 topology_manager.go:138] "Creating topology manager with none policy"
Jul 14 14:30:22.718249 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.718219    2586 container_manager_linux.go:309] "Creating device plugin manager"
Jul 14 14:30:22.718249 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.718227    2586 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 14:30:22.718976 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.718966    2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 14:30:22.720214 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.720204    2586 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 14 14:30:22.723727 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.723715    2586 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 14:30:22.723872 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.723862    2586 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 14 14:30:22.724762 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.724748    2586 kubelet.go:486] "Attempting to sync node with API server"
Jul 14 14:30:22.724821 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.724767    2586 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 14 14:30:22.724821 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.724787    2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 14 14:30:22.724821 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.724799    2586 kubelet.go:398] "Adding apiserver pod source"
Jul 14 14:30:22.724821 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.724808    2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 14 14:30:22.725153 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.725140    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 14:30:22.725187 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.725161    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 14:30:22.728037 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.728016    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9q9kh"
Jul 14 14:30:22.728243 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.728216    2586 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 14 14:30:22.730694 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.730681    2586 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 14 14:30:22.730754 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.730711    2586 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 14 14:30:22.731551 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.731532    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 14 14:30:22.731551 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.731554    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 14 14:30:22.731667 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.731560    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 14 14:30:22.731667 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.731566    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 14 14:30:22.731667 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.731574    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 14 14:30:22.731667 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.731580    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 14 14:30:22.731667 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.731587    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 14 14:30:22.731667 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.731593    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 14 14:30:22.731667 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.731600    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 14 14:30:22.731667 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.731605    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 14 14:30:22.731667 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.731613    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 14 14:30:22.731667 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.731618    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 14 14:30:22.732482 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.732468    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 14 14:30:22.732482 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.732482    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 14 14:30:22.734292 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.734276    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9q9kh"
Jul 14 14:30:22.735468 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.735450    2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-209.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 14 14:30:22.735632 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:22.735614    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 14 14:30:22.735738 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:22.735710    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-209.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 14 14:30:22.736609 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.736594    2586 server.go:1268] "Started kubelet"
Jul 14 14:30:22.736702 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.736676    2586 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 14 14:30:22.737036 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.736758    2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 14 14:30:22.737036 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.736841    2586 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 14 14:30:22.737506 ip-10-0-141-209 systemd[1]: Started Kubernetes Kubelet.
Jul 14 14:30:22.738434 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.738395    2586 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 14 14:30:22.738743 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.738730    2586 server.go:310] "Adding debug handlers to kubelet server"
Jul 14 14:30:22.743035 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:22.741968    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-209.ec2.internal.18c22e67197f00ae  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-209.ec2.internal,UID:ip-10-0-141-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-209.ec2.internal,},FirstTimestamp:2026-07-14 14:30:22.736400558 +0000 UTC m=+0.402586974,LastTimestamp:2026-07-14 14:30:22.736400558 +0000 UTC m=+0.402586974,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-209.ec2.internal,}"
Jul 14 14:30:22.743837 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:22.743818    2586 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 14 14:30:22.747928 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.747914    2586 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 14 14:30:22.747990 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.747932    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 14 14:30:22.748134 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:22.748095    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:22.748763 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.748743    2586 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 14 14:30:22.748839 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.748765    2586 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 14 14:30:22.748839 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.748747    2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 14 14:30:22.748945 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.748843    2586 reconstruct.go:97] "Volume reconstruction finished"
Jul 14 14:30:22.748945 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.748853    2586 reconciler.go:29] "Reconciler: start to sync state"
Jul 14 14:30:22.749040 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.748960    2586 factory.go:55] Registering systemd factory
Jul 14 14:30:22.749040 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.748973    2586 factory.go:223] Registration of the systemd container factory successfully
Jul 14 14:30:22.749244 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.749228    2586 factory.go:153] Registering CRI-O factory
Jul 14 14:30:22.749244 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.749245    2586 factory.go:223] Registration of the crio container factory successfully
Jul 14 14:30:22.749381 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.749289    2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 14 14:30:22.749381 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.749312    2586 factory.go:103] Registering Raw factory
Jul 14 14:30:22.749381 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.749328    2586 manager.go:1196] Started watching for new ooms in manager
Jul 14 14:30:22.749784 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.749772    2586 manager.go:349] Starting recovery of all containers
Jul 14 14:30:22.760172 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.760027    2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 14:30:22.762316 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.762299    2586 manager.go:354] Recovery completed
Jul 14 14:30:22.762606 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:22.762586    2586 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-209.ec2.internal\" not found" node="ip-10-0-141-209.ec2.internal"
Jul 14 14:30:22.766559 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.766547    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 14:30:22.768289 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.768272    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 14:30:22.768344 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.768305    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 14:30:22.768344 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.768322    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasSufficientPID"
Jul 14 14:30:22.768788 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.768776    2586 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 14 14:30:22.768850 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.768789    2586 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 14 14:30:22.768850 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.768792    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 14:30:22.768850 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.768809    2586 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 14:30:22.769513 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.769497    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 14:30:22.769588 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.769528    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 14:30:22.769588 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.769543    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasSufficientPID"
Jul 14 14:30:22.770161 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.770149    2586 policy_none.go:49] "None policy: Start"
Jul 14 14:30:22.770202 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.770165    2586 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 14 14:30:22.770202 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.770175    2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 14 14:30:22.771134 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.771123    2586 policy_none.go:47] "Start"
Jul 14 14:30:22.813228 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.813214    2586 manager.go:337] "Starting Device Plugin manager"
Jul 14 14:30:22.843712 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:22.813245    2586 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 14 14:30:22.843712 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.813255    2586 server.go:87] "Starting device plugin registration server"
Jul 14 14:30:22.843712 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.813524    2586 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 14 14:30:22.843712 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.813537    2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 14 14:30:22.843712 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.813621    2586 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 14 14:30:22.843712 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.813685    2586 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 14 14:30:22.843712 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.813690    2586 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 14 14:30:22.843712 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:22.814228    2586 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 14 14:30:22.843712 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:22.814273    2586 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:22.843712 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.826278    2586 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2622 into the system container \"/system.slice\": <nil>"
Jul 14 14:30:22.868445 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.868390    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 14 14:30:22.869762 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.869746    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 14 14:30:22.869823 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.869796    2586 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 14 14:30:22.869865 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.869825    2586 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 14 14:30:22.869911 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:22.869865    2586 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 14 14:30:22.872373 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.872354    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 14:30:22.914474 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.914372    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 14:30:22.915636 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.915619    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 14:30:22.915741 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.915657    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 14:30:22.915741 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.915671    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasSufficientPID"
Jul 14 14:30:22.915741 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.915700    2586 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-209.ec2.internal"
Jul 14 14:30:22.924329 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.924312    2586 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-209.ec2.internal"
Jul 14 14:30:22.924425 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:22.924337    2586 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-209.ec2.internal\": node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:22.939503 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:22.939477    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:22.970480 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.970454    2586 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-209.ec2.internal"]
Jul 14 14:30:22.970555 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.970525    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 14:30:22.972096 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.972079    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 14:30:22.972175 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.972123    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 14:30:22.972175 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.972134    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasSufficientPID"
Jul 14 14:30:22.973913 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.973892    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 14:30:22.974098 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.974082    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:22.974157 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.974119    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 14:30:22.974685 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.974671    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 14:30:22.974759 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.974691    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 14:30:22.974759 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.974708    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasSufficientPID"
Jul 14 14:30:22.974759 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.974676    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 14:30:22.974848 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.974782    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 14:30:22.974848 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.974792    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasSufficientPID"
Jul 14 14:30:22.975891 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.975872    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:22.975977 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.975898    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 14:30:22.976583 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.976569    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 14:30:22.976662 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.976614    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 14:30:22.976662 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:22.976626    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeHasSufficientPID"
Jul 14 14:30:23.011170 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:23.011147    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-209.ec2.internal\" not found" node="ip-10-0-141-209.ec2.internal"
Jul 14 14:30:23.015359 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:23.015339    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-209.ec2.internal\" not found" node="ip-10-0-141-209.ec2.internal"
Jul 14 14:30:23.039878 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:23.039846    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:23.050767 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.050745    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32e6ac82d25811564f3e9e457bd986bc-config\") pod \"kube-apiserver-proxy-ip-10-0-141-209.ec2.internal\" (UID: \"32e6ac82d25811564f3e9e457bd986bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:23.050869 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.050772    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a154078233a1a53525b6fe59aa34665-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal\" (UID: \"0a154078233a1a53525b6fe59aa34665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:23.050869 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.050789    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a154078233a1a53525b6fe59aa34665-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal\" (UID: \"0a154078233a1a53525b6fe59aa34665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:23.140836 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:23.140808    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:23.151136 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.151116    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32e6ac82d25811564f3e9e457bd986bc-config\") pod \"kube-apiserver-proxy-ip-10-0-141-209.ec2.internal\" (UID: \"32e6ac82d25811564f3e9e457bd986bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:23.151188 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.151140    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a154078233a1a53525b6fe59aa34665-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal\" (UID: \"0a154078233a1a53525b6fe59aa34665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:23.151188 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.151165    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a154078233a1a53525b6fe59aa34665-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal\" (UID: \"0a154078233a1a53525b6fe59aa34665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:23.151252 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.151202    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a154078233a1a53525b6fe59aa34665-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal\" (UID: \"0a154078233a1a53525b6fe59aa34665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:23.151252 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.151215    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32e6ac82d25811564f3e9e457bd986bc-config\") pod \"kube-apiserver-proxy-ip-10-0-141-209.ec2.internal\" (UID: \"32e6ac82d25811564f3e9e457bd986bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:23.151309 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.151243    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a154078233a1a53525b6fe59aa34665-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal\" (UID: \"0a154078233a1a53525b6fe59aa34665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:23.241604 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:23.241553    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:23.313121 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.313083    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:23.318158 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.318142    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:23.341872 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:23.341834    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:23.442366 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:23.442328    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:23.542957 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:23.542878    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:23.643444 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:23.643384    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:23.663244 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.663215    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 14:30:23.672054 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.672032    2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 14 14:30:23.672157 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.672141    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 14:30:23.672198 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.672180    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 14:30:23.672198 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.672189    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 14:30:23.740497 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.740447    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 14:25:23 +0000 UTC" deadline="2028-04-22 03:07:21.833249644 +0000 UTC"
Jul 14 14:30:23.740497 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:23.740492    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15540h36m58.092760134s"
Jul 14 14:30:23.743626 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:23.743614    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:23.844369 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:23.844277    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:23.945198 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:23.945147    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:24.045728 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:24.045694    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-209.ec2.internal\" not found"
Jul 14 14:30:24.076890 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.076874    2586 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 14:30:24.148214 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.148152    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:24.160130 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.160105    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 14:30:24.161352 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.161337    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-209.ec2.internal"
Jul 14 14:30:24.170729 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.170705    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 14:30:24.371585 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:24.371548    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e6ac82d25811564f3e9e457bd986bc.slice/crio-82e75e9835b3e7b2208fc584b1caa78f46845dce284b6212e98bb444afeea6a0 WatchSource:0}: Error finding container 82e75e9835b3e7b2208fc584b1caa78f46845dce284b6212e98bb444afeea6a0: Status 404 returned error can't find the container with id 82e75e9835b3e7b2208fc584b1caa78f46845dce284b6212e98bb444afeea6a0
Jul 14 14:30:24.375786 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.375769    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 14:30:24.680847 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.680818    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 14:30:24.728568 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.728535    2586 apiserver.go:52] "Watching apiserver"
Jul 14 14:30:24.738023 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.738000    2586 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 14 14:30:24.739802 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.739777    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-fl6rb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal","openshift-multus/multus-additional-cni-plugins-nj27g","openshift-network-diagnostics/network-check-target-84g4n","openshift-network-operator/iptables-alerter-nwmln","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65","openshift-cluster-node-tuning-operator/tuned-dh9dt","openshift-image-registry/node-ca-nqtwh","openshift-multus/multus-kkp8m","openshift-multus/network-metrics-daemon-zqd7b","openshift-ovn-kubernetes/ovnkube-node-x8sbr","kube-system/global-pull-secret-syncer-rlbbv","kube-system/konnectivity-agent-t84p9","kube-system/kube-apiserver-proxy-ip-10-0-141-209.ec2.internal"]
Jul 14 14:30:24.742090 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.742074    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.743317 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.743298    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.744798 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.744775    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.745401 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.745383    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wg7cq\""
Jul 14 14:30:24.745401 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.745396    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 14 14:30:24.745540 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.745396    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 14 14:30:24.745904 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.745883    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:24.745999 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:24.745982    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:24.746340 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.746318    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 14 14:30:24.746449 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.746383    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 14 14:30:24.747123 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.747104    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nwmln"
Jul 14 14:30:24.748008 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.747989    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 14 14:30:24.748117 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.748045    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 14 14:30:24.748183 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.748168    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 14 14:30:24.748183 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.748179    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rv8xm\""
Jul 14 14:30:24.748287 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.748269    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 14 14:30:24.748507 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.748474    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 14 14:30:24.748507 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.748496    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.748696 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.748499    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 14 14:30:24.748696 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.748561    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 14 14:30:24.752628 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.749617    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 14 14:30:24.752628 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.749649    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 14 14:30:24.752628 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.750192    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 14 14:30:24.752628 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.750726    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 14 14:30:24.752628 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.751057    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8jbh8\""
Jul 14 14:30:24.752628 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.751405    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 14 14:30:24.752628 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.751688    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 14 14:30:24.752628 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.752010    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.752628 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.752024    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7bn78\""
Jul 14 14:30:24.752628 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.752344    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 14 14:30:24.753275 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.752789    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 14 14:30:24.753275 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.752791    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-htgkv\""
Jul 14 14:30:24.753275 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.753037    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fl6rb"
Jul 14 14:30:24.753513 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.753454    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 14 14:30:24.754643 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.754611    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 14 14:30:24.754740 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.754697    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:24.754811 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:24.754771    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:24.755188 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.755171    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-blvss\""
Jul 14 14:30:24.755790 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.755774    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 14 14:30:24.756092 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.756077    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t84p9"
Jul 14 14:30:24.756904 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.756887    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-c45lb\""
Jul 14 14:30:24.757250 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.757234    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 14 14:30:24.757484 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.757471    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:24.757557 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:24.757538    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:24.758746 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.758716    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 14 14:30:24.758846 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.758790    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 14 14:30:24.758846 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.758809    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nqtwh"
Jul 14 14:30:24.758960 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.758891    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wpl8p\""
Jul 14 14:30:24.759442 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759417    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-kubelet\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.759519 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759478    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-system-cni-dir\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.759519 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759506    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-run-k8s-cni-cncf-io\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.759607 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759541    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-systemd\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.759607 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759590    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-tuned\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.759682 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759639    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-run-systemd\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.759733 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759678    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-hostroot\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.759733 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759705    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z57ph\" (UniqueName: \"kubernetes.io/projected/401faa4d-42da-4c1c-af1c-fe150089dfe0-kube-api-access-z57ph\") pod \"node-resolver-fl6rb\" (UID: \"401faa4d-42da-4c1c-af1c-fe150089dfe0\") " pod="openshift-dns/node-resolver-fl6rb"
Jul 14 14:30:24.759812 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759741    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5a8692c5-50c8-4012-8415-6d580d7ca0bb-iptables-alerter-script\") pod \"iptables-alerter-nwmln\" (UID: \"5a8692c5-50c8-4012-8415-6d580d7ca0bb\") " pod="openshift-network-operator/iptables-alerter-nwmln"
Jul 14 14:30:24.759812 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759767    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-slash\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.759812 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759791    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-cni-netd\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.759957 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759815    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-var-lib-kubelet\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.759957 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759840    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76945752-395d-4809-abb1-cd8e68793179-cni-binary-copy\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.759957 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759864    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bs2c\" (UniqueName: \"kubernetes.io/projected/76945752-395d-4809-abb1-cd8e68793179-kube-api-access-6bs2c\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.759957 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759891    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-run\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.759957 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759921    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e7ba746-3e33-4f52-bd37-1e2d9242b207-tmp\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.760218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.759964    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-run-ovn-kubernetes\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.760218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760002    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.760218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760031    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76945752-395d-4809-abb1-cd8e68793179-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.760218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760063    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-registration-dir\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.760218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760091    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-kubernetes\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.760218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760108    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-sys\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.760218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760130    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-run-openvswitch\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.760218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760147    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-run-ovn\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.760218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760162    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-etc-selinux\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.760218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760192    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-modprobe-d\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.760218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760223    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-lib-modules\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760251    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-ovnkube-script-lib\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760359    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-multus-socket-dir-parent\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760379    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-var-lib-cni-bin\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760397    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-multus-conf-dir\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760411    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f1cc50dd-47f7-4a20-8047-0cef69221ee1-multus-daemon-config\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760449    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401faa4d-42da-4c1c-af1c-fe150089dfe0-tmp-dir\") pod \"node-resolver-fl6rb\" (UID: \"401faa4d-42da-4c1c-af1c-fe150089dfe0\") " pod="openshift-dns/node-resolver-fl6rb"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760475    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-cni-bin\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760511    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-env-overrides\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760531    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76945752-395d-4809-abb1-cd8e68793179-system-cni-dir\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760579    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7q67\" (UniqueName: \"kubernetes.io/projected/5a8692c5-50c8-4012-8415-6d580d7ca0bb-kube-api-access-c7q67\") pod \"iptables-alerter-nwmln\" (UID: \"5a8692c5-50c8-4012-8415-6d580d7ca0bb\") " pod="openshift-network-operator/iptables-alerter-nwmln"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760611    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-socket-dir\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760635    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-device-dir\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760663    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9n4n\" (UniqueName: \"kubernetes.io/projected/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-kube-api-access-w9n4n\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760695    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-etc-openvswitch\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760722    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-run-netns\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.761081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760744    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/401faa4d-42da-4c1c-af1c-fe150089dfe0-hosts-file\") pod \"node-resolver-fl6rb\" (UID: \"401faa4d-42da-4c1c-af1c-fe150089dfe0\") " pod="openshift-dns/node-resolver-fl6rb"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760768    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76945752-395d-4809-abb1-cd8e68793179-cnibin\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760816    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76945752-395d-4809-abb1-cd8e68793179-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760837    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-sysconfig\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760852    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1cc50dd-47f7-4a20-8047-0cef69221ee1-cni-binary-copy\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760866    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76945752-395d-4809-abb1-cd8e68793179-os-release\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760881    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-var-lib-kubelet\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760917    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-host\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760933    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-node-log\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760948    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76945752-395d-4809-abb1-cd8e68793179-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.760970    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4gmw\" (UniqueName: \"kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw\") pod \"network-check-target-84g4n\" (UID: \"667c8f62-f765-4395-9740-d0c0bc87102c\") " pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761024    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-systemd-units\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761063    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-cnibin\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761094    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-var-lib-cni-multus\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761124    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-sysctl-d\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761151    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-sysctl-conf\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.761961 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761178    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-run-netns\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761203    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-ovnkube-config\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761227    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z4bz\" (UniqueName: \"kubernetes.io/projected/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-kube-api-access-2z4bz\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761251    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-multus-cni-dir\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761275    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-os-release\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761313    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-etc-kubernetes\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761341    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-var-lib-openvswitch\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761364    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-log-socket\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761393    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-ovn-node-metrics-cert\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761394    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761464    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-run-multus-certs\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761488    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a8692c5-50c8-4012-8415-6d580d7ca0bb-host-slash\") pod \"iptables-alerter-nwmln\" (UID: \"5a8692c5-50c8-4012-8415-6d580d7ca0bb\") " pod="openshift-network-operator/iptables-alerter-nwmln"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761506    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761546    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-sys-fs\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761562    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pdgz2\""
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761566    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lccc7\" (UniqueName: \"kubernetes.io/projected/9e7ba746-3e33-4f52-bd37-1e2d9242b207-kube-api-access-lccc7\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761618    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77rvq\" (UniqueName: \"kubernetes.io/projected/f1cc50dd-47f7-4a20-8047-0cef69221ee1-kube-api-access-77rvq\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761630    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 14 14:30:24.762645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.761642    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 14 14:30:24.764514 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.764493    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 14:30:24.787729 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.787712    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s46jd"
Jul 14 14:30:24.795872 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.795851    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s46jd"
Jul 14 14:30:24.850276 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.850239    2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 14 14:30:24.859831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.859807    2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 14:30:24.862038 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862008    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3febdf7d-4462-4194-a512-7c860fc44797-kubelet-config\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:24.862149 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862048    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-var-lib-kubelet\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.862149 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862074    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-host\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.862231 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862147    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-var-lib-kubelet\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.862389 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862113    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-node-log\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862194    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-node-log\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862320    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-host\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862674    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76945752-395d-4809-abb1-cd8e68793179-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862725    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gmw\" (UniqueName: \"kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw\") pod \"network-check-target-84g4n\" (UID: \"667c8f62-f765-4395-9740-d0c0bc87102c\") " pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862761    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-systemd-units\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862811    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-cnibin\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862844    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-var-lib-cni-multus\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862875    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74295487-4d63-42cb-b367-fda447a5217b-host\") pod \"node-ca-nqtwh\" (UID: \"74295487-4d63-42cb-b367-fda447a5217b\") " pod="openshift-image-registry/node-ca-nqtwh"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862923    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs\") pod \"network-metrics-daemon-zqd7b\" (UID: \"85b30a97-2ff1-4091-9e77-4359a1c4c242\") " pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862959    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3febdf7d-4462-4194-a512-7c860fc44797-dbus\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.862985    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-sysctl-d\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863013    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-sysctl-conf\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863043    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-run-netns\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863074    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-ovnkube-config\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863109    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z4bz\" (UniqueName: \"kubernetes.io/projected/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-kube-api-access-2z4bz\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863140    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-multus-cni-dir\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.865057 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863169    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-os-release\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863194    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-etc-kubernetes\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863229    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-var-lib-openvswitch\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863260    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-log-socket\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863341    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-ovn-node-metrics-cert\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863375    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-run-multus-certs\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863419    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a8692c5-50c8-4012-8415-6d580d7ca0bb-host-slash\") pod \"iptables-alerter-nwmln\" (UID: \"5a8692c5-50c8-4012-8415-6d580d7ca0bb\") " pod="openshift-network-operator/iptables-alerter-nwmln"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863567    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863599    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-sys-fs\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863632    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmd6r\" (UniqueName: \"kubernetes.io/projected/85b30a97-2ff1-4091-9e77-4359a1c4c242-kube-api-access-dmd6r\") pod \"network-metrics-daemon-zqd7b\" (UID: \"85b30a97-2ff1-4091-9e77-4359a1c4c242\") " pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863679    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lccc7\" (UniqueName: \"kubernetes.io/projected/9e7ba746-3e33-4f52-bd37-1e2d9242b207-kube-api-access-lccc7\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863706    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77rvq\" (UniqueName: \"kubernetes.io/projected/f1cc50dd-47f7-4a20-8047-0cef69221ee1-kube-api-access-77rvq\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863772    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-kubelet\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863803    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-system-cni-dir\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863834    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-run-k8s-cni-cncf-io\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863868    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-systemd\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863892    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-tuned\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.865955 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.863964    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-run-systemd\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864015    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-hostroot\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864110    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z57ph\" (UniqueName: \"kubernetes.io/projected/401faa4d-42da-4c1c-af1c-fe150089dfe0-kube-api-access-z57ph\") pod \"node-resolver-fl6rb\" (UID: \"401faa4d-42da-4c1c-af1c-fe150089dfe0\") " pod="openshift-dns/node-resolver-fl6rb"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864153    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5a8692c5-50c8-4012-8415-6d580d7ca0bb-iptables-alerter-script\") pod \"iptables-alerter-nwmln\" (UID: \"5a8692c5-50c8-4012-8415-6d580d7ca0bb\") " pod="openshift-network-operator/iptables-alerter-nwmln"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864193    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-slash\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864236    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-cni-netd\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864362    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-var-lib-kubelet\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864404    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76945752-395d-4809-abb1-cd8e68793179-cni-binary-copy\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864437    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bs2c\" (UniqueName: \"kubernetes.io/projected/76945752-395d-4809-abb1-cd8e68793179-kube-api-access-6bs2c\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864467    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-run\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864510    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e7ba746-3e33-4f52-bd37-1e2d9242b207-tmp\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864542    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-run-ovn-kubernetes\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864598    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864636    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76945752-395d-4809-abb1-cd8e68793179-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864679    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-registration-dir\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864703    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-kubernetes\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864732    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-sys\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.866831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864771    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-run-openvswitch\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864800    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-run-ovn\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864826    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f12e6b3b-d351-4faf-bf16-9ccfd4100848-agent-certs\") pod \"konnectivity-agent-t84p9\" (UID: \"f12e6b3b-d351-4faf-bf16-9ccfd4100848\") " pod="kube-system/konnectivity-agent-t84p9"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864857    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-etc-selinux\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.864885    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-modprobe-d\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.865069    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-run-systemd\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.865170    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-kubelet\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.865156    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-lib-modules\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.865215    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-lib-modules\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.865401    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-systemd-units\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.865573    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-log-socket\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.865573    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-run-netns\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.865669    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-modprobe-d\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.865801    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-run-ovn\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.865851    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-run-ovn-kubernetes\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.865904    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-cnibin\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.866269    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-run-k8s-cni-cncf-io\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.866371    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-var-lib-cni-multus\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.867717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.866571    2586 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.866659    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76945752-395d-4809-abb1-cd8e68793179-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.866679    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5a8692c5-50c8-4012-8415-6d580d7ca0bb-iptables-alerter-script\") pod \"iptables-alerter-nwmln\" (UID: \"5a8692c5-50c8-4012-8415-6d580d7ca0bb\") " pod="openshift-network-operator/iptables-alerter-nwmln"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.866686    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-run\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.866849    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76945752-395d-4809-abb1-cd8e68793179-cni-binary-copy\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.866950    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-systemd\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.866969    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-sysctl-d\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.866976    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-sysctl-conf\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.866996    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-ovnkube-script-lib\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867064    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-slash\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867081    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-multus-socket-dir-parent\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867140    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-cni-netd\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867157    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-var-lib-cni-bin\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867248    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-var-lib-kubelet\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867353    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-registration-dir\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867489    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-ovnkube-config\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867579    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-kubernetes\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.868339 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867592    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-ovnkube-script-lib\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867584    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a8692c5-50c8-4012-8415-6d580d7ca0bb-host-slash\") pod \"iptables-alerter-nwmln\" (UID: \"5a8692c5-50c8-4012-8415-6d580d7ca0bb\") " pod="openshift-network-operator/iptables-alerter-nwmln"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867695    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-sys\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867778    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867862    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-run-multus-certs\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867897    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-os-release\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867912    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-sys-fs\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.867952    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-var-lib-openvswitch\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868426    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-etc-kubernetes\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868473    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-etc-selinux\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868490    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-hostroot\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868507    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-system-cni-dir\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868526    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868573    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-multus-conf-dir\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868601    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-multus-conf-dir\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868621    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-multus-socket-dir-parent\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868671    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-var-lib-cni-bin\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868604    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f1cc50dd-47f7-4a20-8047-0cef69221ee1-multus-daemon-config\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.869139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868714    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401faa4d-42da-4c1c-af1c-fe150089dfe0-tmp-dir\") pod \"node-resolver-fl6rb\" (UID: \"401faa4d-42da-4c1c-af1c-fe150089dfe0\") " pod="openshift-dns/node-resolver-fl6rb"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868743    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-cni-bin\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868773    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-env-overrides\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868800    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76945752-395d-4809-abb1-cd8e68793179-system-cni-dir\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868825    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7q67\" (UniqueName: \"kubernetes.io/projected/5a8692c5-50c8-4012-8415-6d580d7ca0bb-kube-api-access-c7q67\") pod \"iptables-alerter-nwmln\" (UID: \"5a8692c5-50c8-4012-8415-6d580d7ca0bb\") " pod="openshift-network-operator/iptables-alerter-nwmln"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868851    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-socket-dir\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868876    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-device-dir\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868900    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9n4n\" (UniqueName: \"kubernetes.io/projected/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-kube-api-access-w9n4n\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868959    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-etc-openvswitch\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.868985    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-run-netns\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869025    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/401faa4d-42da-4c1c-af1c-fe150089dfe0-hosts-file\") pod \"node-resolver-fl6rb\" (UID: \"401faa4d-42da-4c1c-af1c-fe150089dfe0\") " pod="openshift-dns/node-resolver-fl6rb"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869050    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76945752-395d-4809-abb1-cd8e68793179-cnibin\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869077    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76945752-395d-4809-abb1-cd8e68793179-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869107    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869124    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f1cc50dd-47f7-4a20-8047-0cef69221ee1-multus-daemon-config\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869136    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-sysconfig\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869178    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1cc50dd-47f7-4a20-8047-0cef69221ee1-cni-binary-copy\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.870106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869181    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-socket-dir\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869185    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/401faa4d-42da-4c1c-af1c-fe150089dfe0-hosts-file\") pod \"node-resolver-fl6rb\" (UID: \"401faa4d-42da-4c1c-af1c-fe150089dfe0\") " pod="openshift-dns/node-resolver-fl6rb"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869253    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76945752-395d-4809-abb1-cd8e68793179-cnibin\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869278    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76945752-395d-4809-abb1-cd8e68793179-os-release\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869335    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-sysconfig\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869383    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-etc-openvswitch\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869423    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-host-run-netns\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869499    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1cc50dd-47f7-4a20-8047-0cef69221ee1-multus-cni-dir\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869206    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76945752-395d-4809-abb1-cd8e68793179-os-release\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869561    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f12e6b3b-d351-4faf-bf16-9ccfd4100848-konnectivity-ca\") pod \"konnectivity-agent-t84p9\" (UID: \"f12e6b3b-d351-4faf-bf16-9ccfd4100848\") " pod="kube-system/konnectivity-agent-t84p9"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869601    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76945752-395d-4809-abb1-cd8e68793179-system-cni-dir\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869610    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-host-cni-bin\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869593    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/74295487-4d63-42cb-b367-fda447a5217b-serviceca\") pod \"node-ca-nqtwh\" (UID: \"74295487-4d63-42cb-b367-fda447a5217b\") " pod="openshift-image-registry/node-ca-nqtwh"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869645    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-run-openvswitch\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869660    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx4w5\" (UniqueName: \"kubernetes.io/projected/74295487-4d63-42cb-b367-fda447a5217b-kube-api-access-nx4w5\") pod \"node-ca-nqtwh\" (UID: \"74295487-4d63-42cb-b367-fda447a5217b\") " pod="openshift-image-registry/node-ca-nqtwh"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869665    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-device-dir\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869549    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-env-overrides\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.870771 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869769    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1cc50dd-47f7-4a20-8047-0cef69221ee1-cni-binary-copy\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.871328 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869860    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76945752-395d-4809-abb1-cd8e68793179-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.871328 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.869952    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401faa4d-42da-4c1c-af1c-fe150089dfe0-tmp-dir\") pod \"node-resolver-fl6rb\" (UID: \"401faa4d-42da-4c1c-af1c-fe150089dfe0\") " pod="openshift-dns/node-resolver-fl6rb"
Jul 14 14:30:24.871328 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.870223    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76945752-395d-4809-abb1-cd8e68793179-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.871328 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.870242    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e7ba746-3e33-4f52-bd37-1e2d9242b207-etc-tuned\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.871328 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.870469    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e7ba746-3e33-4f52-bd37-1e2d9242b207-tmp\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.871698 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.871679    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-ovn-node-metrics-cert\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.872666 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:24.872648    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 14:30:24.872666 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:24.872672    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 14:30:24.872831 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:24.872686    2586 projected.go:196] Error preparing data for projected volume kube-api-access-d4gmw for pod openshift-network-diagnostics/network-check-target-84g4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 14:30:24.872831 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:24.872755    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw podName:667c8f62-f765-4395-9740-d0c0bc87102c nodeName:}" failed. No retries permitted until 2026-07-14 14:30:25.372734854 +0000 UTC m=+3.038921269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d4gmw" (UniqueName: "kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw") pod "network-check-target-84g4n" (UID: "667c8f62-f765-4395-9740-d0c0bc87102c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 14:30:24.874261 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.874236    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bs2c\" (UniqueName: \"kubernetes.io/projected/76945752-395d-4809-abb1-cd8e68793179-kube-api-access-6bs2c\") pod \"multus-additional-cni-plugins-nj27g\" (UID: \"76945752-395d-4809-abb1-cd8e68793179\") " pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:24.874713 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.874661    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77rvq\" (UniqueName: \"kubernetes.io/projected/f1cc50dd-47f7-4a20-8047-0cef69221ee1-kube-api-access-77rvq\") pod \"multus-kkp8m\" (UID: \"f1cc50dd-47f7-4a20-8047-0cef69221ee1\") " pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:24.876726 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.876676    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-209.ec2.internal" event={"ID":"32e6ac82d25811564f3e9e457bd986bc","Type":"ContainerStarted","Data":"82e75e9835b3e7b2208fc584b1caa78f46845dce284b6212e98bb444afeea6a0"}
Jul 14 14:30:24.877040 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.877021    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z57ph\" (UniqueName: \"kubernetes.io/projected/401faa4d-42da-4c1c-af1c-fe150089dfe0-kube-api-access-z57ph\") pod \"node-resolver-fl6rb\" (UID: \"401faa4d-42da-4c1c-af1c-fe150089dfe0\") " pod="openshift-dns/node-resolver-fl6rb"
Jul 14 14:30:24.877083 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.877018    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z4bz\" (UniqueName: \"kubernetes.io/projected/a61c2cbb-e090-4ca1-825b-e0b85151fcf2-kube-api-access-2z4bz\") pod \"ovnkube-node-x8sbr\" (UID: \"a61c2cbb-e090-4ca1-825b-e0b85151fcf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:24.877674 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.877648    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lccc7\" (UniqueName: \"kubernetes.io/projected/9e7ba746-3e33-4f52-bd37-1e2d9242b207-kube-api-access-lccc7\") pod \"tuned-dh9dt\" (UID: \"9e7ba746-3e33-4f52-bd37-1e2d9242b207\") " pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:24.877785 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.877765    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9n4n\" (UniqueName: \"kubernetes.io/projected/c91bb9dd-9ad3-4d3d-8555-b5f61c681f55-kube-api-access-w9n4n\") pod \"aws-ebs-csi-driver-node-t4b65\" (UID: \"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:24.877890 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.877876    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7q67\" (UniqueName: \"kubernetes.io/projected/5a8692c5-50c8-4012-8415-6d580d7ca0bb-kube-api-access-c7q67\") pod \"iptables-alerter-nwmln\" (UID: \"5a8692c5-50c8-4012-8415-6d580d7ca0bb\") " pod="openshift-network-operator/iptables-alerter-nwmln"
Jul 14 14:30:24.970084 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.969998    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmd6r\" (UniqueName: \"kubernetes.io/projected/85b30a97-2ff1-4091-9e77-4359a1c4c242-kube-api-access-dmd6r\") pod \"network-metrics-daemon-zqd7b\" (UID: \"85b30a97-2ff1-4091-9e77-4359a1c4c242\") " pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:24.970084 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.970037    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f12e6b3b-d351-4faf-bf16-9ccfd4100848-agent-certs\") pod \"konnectivity-agent-t84p9\" (UID: \"f12e6b3b-d351-4faf-bf16-9ccfd4100848\") " pod="kube-system/konnectivity-agent-t84p9"
Jul 14 14:30:24.970084 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.970069    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:24.970347 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.970099    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f12e6b3b-d351-4faf-bf16-9ccfd4100848-konnectivity-ca\") pod \"konnectivity-agent-t84p9\" (UID: \"f12e6b3b-d351-4faf-bf16-9ccfd4100848\") " pod="kube-system/konnectivity-agent-t84p9"
Jul 14 14:30:24.970347 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.970123    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/74295487-4d63-42cb-b367-fda447a5217b-serviceca\") pod \"node-ca-nqtwh\" (UID: \"74295487-4d63-42cb-b367-fda447a5217b\") " pod="openshift-image-registry/node-ca-nqtwh"
Jul 14 14:30:24.970347 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.970146    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx4w5\" (UniqueName: \"kubernetes.io/projected/74295487-4d63-42cb-b367-fda447a5217b-kube-api-access-nx4w5\") pod \"node-ca-nqtwh\" (UID: \"74295487-4d63-42cb-b367-fda447a5217b\") " pod="openshift-image-registry/node-ca-nqtwh"
Jul 14 14:30:24.970347 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.970169    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3febdf7d-4462-4194-a512-7c860fc44797-kubelet-config\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:24.970347 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:24.970172    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 14:30:24.970347 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.970215    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74295487-4d63-42cb-b367-fda447a5217b-host\") pod \"node-ca-nqtwh\" (UID: \"74295487-4d63-42cb-b367-fda447a5217b\") " pod="openshift-image-registry/node-ca-nqtwh"
Jul 14 14:30:24.970347 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:24.970259    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret podName:3febdf7d-4462-4194-a512-7c860fc44797 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:25.470239254 +0000 UTC m=+3.136425676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret") pod "global-pull-secret-syncer-rlbbv" (UID: "3febdf7d-4462-4194-a512-7c860fc44797") : object "kube-system"/"original-pull-secret" not registered
Jul 14 14:30:24.970347 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.970306    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3febdf7d-4462-4194-a512-7c860fc44797-kubelet-config\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:24.970347 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.970345    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs\") pod \"network-metrics-daemon-zqd7b\" (UID: \"85b30a97-2ff1-4091-9e77-4359a1c4c242\") " pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:24.970907 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.970392    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3febdf7d-4462-4194-a512-7c860fc44797-dbus\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:24.970907 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.970431    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74295487-4d63-42cb-b367-fda447a5217b-host\") pod \"node-ca-nqtwh\" (UID: \"74295487-4d63-42cb-b367-fda447a5217b\") " pod="openshift-image-registry/node-ca-nqtwh"
Jul 14 14:30:24.970907 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:24.970474    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 14:30:24.970907 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:24.970540    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs podName:85b30a97-2ff1-4091-9e77-4359a1c4c242 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:25.470525564 +0000 UTC m=+3.136711966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs") pod "network-metrics-daemon-zqd7b" (UID: "85b30a97-2ff1-4091-9e77-4359a1c4c242") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 14:30:24.970907 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.970576    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3febdf7d-4462-4194-a512-7c860fc44797-dbus\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:24.970907 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.970719    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/74295487-4d63-42cb-b367-fda447a5217b-serviceca\") pod \"node-ca-nqtwh\" (UID: \"74295487-4d63-42cb-b367-fda447a5217b\") " pod="openshift-image-registry/node-ca-nqtwh"
Jul 14 14:30:24.970907 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.970847    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f12e6b3b-d351-4faf-bf16-9ccfd4100848-konnectivity-ca\") pod \"konnectivity-agent-t84p9\" (UID: \"f12e6b3b-d351-4faf-bf16-9ccfd4100848\") " pod="kube-system/konnectivity-agent-t84p9"
Jul 14 14:30:24.972662 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.972643    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f12e6b3b-d351-4faf-bf16-9ccfd4100848-agent-certs\") pod \"konnectivity-agent-t84p9\" (UID: \"f12e6b3b-d351-4faf-bf16-9ccfd4100848\") " pod="kube-system/konnectivity-agent-t84p9"
Jul 14 14:30:24.981768 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.981738    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx4w5\" (UniqueName: \"kubernetes.io/projected/74295487-4d63-42cb-b367-fda447a5217b-kube-api-access-nx4w5\") pod \"node-ca-nqtwh\" (UID: \"74295487-4d63-42cb-b367-fda447a5217b\") " pod="openshift-image-registry/node-ca-nqtwh"
Jul 14 14:30:24.981923 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:24.981904    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmd6r\" (UniqueName: \"kubernetes.io/projected/85b30a97-2ff1-4091-9e77-4359a1c4c242-kube-api-access-dmd6r\") pod \"network-metrics-daemon-zqd7b\" (UID: \"85b30a97-2ff1-4091-9e77-4359a1c4c242\") " pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:25.019904 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:25.019870    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a154078233a1a53525b6fe59aa34665.slice/crio-c5761f541c40d69ab75252f89e40cab08da863ed0015a7f47f36acbb223e273a WatchSource:0}: Error finding container c5761f541c40d69ab75252f89e40cab08da863ed0015a7f47f36acbb223e273a: Status 404 returned error can't find the container with id c5761f541c40d69ab75252f89e40cab08da863ed0015a7f47f36acbb223e273a
Jul 14 14:30:25.056812 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.056773    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dh9dt"
Jul 14 14:30:25.063493 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.063466    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:25.064568 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:25.064546    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7ba746_3e33_4f52_bd37_1e2d9242b207.slice/crio-ef7d5508230342ae8425e9c304ef420a1cc2025dbabf66d08180013f35f1cb25 WatchSource:0}: Error finding container ef7d5508230342ae8425e9c304ef420a1cc2025dbabf66d08180013f35f1cb25: Status 404 returned error can't find the container with id ef7d5508230342ae8425e9c304ef420a1cc2025dbabf66d08180013f35f1cb25
Jul 14 14:30:25.070637 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.070617    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nj27g"
Jul 14 14:30:25.070944 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:25.070924    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61c2cbb_e090_4ca1_825b_e0b85151fcf2.slice/crio-5e45cf6ebdb6cc8da423503b3db763b567f78d127df3673ba496063882d46e19 WatchSource:0}: Error finding container 5e45cf6ebdb6cc8da423503b3db763b567f78d127df3673ba496063882d46e19: Status 404 returned error can't find the container with id 5e45cf6ebdb6cc8da423503b3db763b567f78d127df3673ba496063882d46e19
Jul 14 14:30:25.075530 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.075276    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nwmln"
Jul 14 14:30:25.078656 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:25.078634    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76945752_395d_4809_abb1_cd8e68793179.slice/crio-ebec0ad070a690ab281c3901471b41c8fedd6edca20f8b50dcf94cb12b25614d WatchSource:0}: Error finding container ebec0ad070a690ab281c3901471b41c8fedd6edca20f8b50dcf94cb12b25614d: Status 404 returned error can't find the container with id ebec0ad070a690ab281c3901471b41c8fedd6edca20f8b50dcf94cb12b25614d
Jul 14 14:30:25.080476 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.080459    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65"
Jul 14 14:30:25.084592 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:25.084571    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8692c5_50c8_4012_8415_6d580d7ca0bb.slice/crio-d0758389ba96cdfce4c9bf4370541e766e412744e760a4b3dab576a7baf691f7 WatchSource:0}: Error finding container d0758389ba96cdfce4c9bf4370541e766e412744e760a4b3dab576a7baf691f7: Status 404 returned error can't find the container with id d0758389ba96cdfce4c9bf4370541e766e412744e760a4b3dab576a7baf691f7
Jul 14 14:30:25.086105 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.086087    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kkp8m"
Jul 14 14:30:25.089220 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:25.089195    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91bb9dd_9ad3_4d3d_8555_b5f61c681f55.slice/crio-6290c7296b06489a7ba50035bae435b5257f3923a5ef2f34d7ddd78ddfd8e7e7 WatchSource:0}: Error finding container 6290c7296b06489a7ba50035bae435b5257f3923a5ef2f34d7ddd78ddfd8e7e7: Status 404 returned error can't find the container with id 6290c7296b06489a7ba50035bae435b5257f3923a5ef2f34d7ddd78ddfd8e7e7
Jul 14 14:30:25.091962 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.091899    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fl6rb"
Jul 14 14:30:25.094308 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:25.094287    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1cc50dd_47f7_4a20_8047_0cef69221ee1.slice/crio-d8e2746ad8a1235b72b421aa1a26e624284268ebfd934039e1cf34bc6c6dad84 WatchSource:0}: Error finding container d8e2746ad8a1235b72b421aa1a26e624284268ebfd934039e1cf34bc6c6dad84: Status 404 returned error can't find the container with id d8e2746ad8a1235b72b421aa1a26e624284268ebfd934039e1cf34bc6c6dad84
Jul 14 14:30:25.098303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.098282    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t84p9"
Jul 14 14:30:25.100154 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:25.100131    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401faa4d_42da_4c1c_af1c_fe150089dfe0.slice/crio-ee64455f421f3a0066a2d5e939cf627c6ffa35610c9d983ded73a291eb7fc0a8 WatchSource:0}: Error finding container ee64455f421f3a0066a2d5e939cf627c6ffa35610c9d983ded73a291eb7fc0a8: Status 404 returned error can't find the container with id ee64455f421f3a0066a2d5e939cf627c6ffa35610c9d983ded73a291eb7fc0a8
Jul 14 14:30:25.103316 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.103298    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nqtwh"
Jul 14 14:30:25.107645 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:25.107626    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12e6b3b_d351_4faf_bf16_9ccfd4100848.slice/crio-214882e7e703185cab5c94bd1ac5a1cb0b66ab485e74043268dcdf6d26de26f6 WatchSource:0}: Error finding container 214882e7e703185cab5c94bd1ac5a1cb0b66ab485e74043268dcdf6d26de26f6: Status 404 returned error can't find the container with id 214882e7e703185cab5c94bd1ac5a1cb0b66ab485e74043268dcdf6d26de26f6
Jul 14 14:30:25.112172 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:30:25.112148    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74295487_4d63_42cb_b367_fda447a5217b.slice/crio-cab4966aace5341901eb4903778c83993d43cc6238a8ec20ffc893548395a957 WatchSource:0}: Error finding container cab4966aace5341901eb4903778c83993d43cc6238a8ec20ffc893548395a957: Status 404 returned error can't find the container with id cab4966aace5341901eb4903778c83993d43cc6238a8ec20ffc893548395a957
Jul 14 14:30:25.376126 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.373973    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gmw\" (UniqueName: \"kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw\") pod \"network-check-target-84g4n\" (UID: \"667c8f62-f765-4395-9740-d0c0bc87102c\") " pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:25.376126 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:25.374289    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 14:30:25.376126 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:25.374312    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 14:30:25.376126 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:25.374325    2586 projected.go:196] Error preparing data for projected volume kube-api-access-d4gmw for pod openshift-network-diagnostics/network-check-target-84g4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 14:30:25.376126 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:25.374393    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw podName:667c8f62-f765-4395-9740-d0c0bc87102c nodeName:}" failed. No retries permitted until 2026-07-14 14:30:26.374373106 +0000 UTC m=+4.040559539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4gmw" (UniqueName: "kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw") pod "network-check-target-84g4n" (UID: "667c8f62-f765-4395-9740-d0c0bc87102c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 14:30:25.474530 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.474496    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:25.474703 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.474552    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs\") pod \"network-metrics-daemon-zqd7b\" (UID: \"85b30a97-2ff1-4091-9e77-4359a1c4c242\") " pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:25.474703 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:25.474662    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 14:30:25.474703 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:25.474672    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 14:30:25.474892 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:25.474714    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs podName:85b30a97-2ff1-4091-9e77-4359a1c4c242 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:26.474700278 +0000 UTC m=+4.140886679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs") pod "network-metrics-daemon-zqd7b" (UID: "85b30a97-2ff1-4091-9e77-4359a1c4c242") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 14:30:25.474892 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:25.474740    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret podName:3febdf7d-4462-4194-a512-7c860fc44797 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:26.474720301 +0000 UTC m=+4.140906720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret") pod "global-pull-secret-syncer-rlbbv" (UID: "3febdf7d-4462-4194-a512-7c860fc44797") : object "kube-system"/"original-pull-secret" not registered
Jul 14 14:30:25.798331 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.798261    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 14:25:25 +0000 UTC" deadline="2027-12-28 20:03:24.366968285 +0000 UTC"
Jul 14 14:30:25.798331 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.798328    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12773h32m58.568644898s"
Jul 14 14:30:25.882159 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.882116    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t84p9" event={"ID":"f12e6b3b-d351-4faf-bf16-9ccfd4100848","Type":"ContainerStarted","Data":"214882e7e703185cab5c94bd1ac5a1cb0b66ab485e74043268dcdf6d26de26f6"}
Jul 14 14:30:25.888638 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.888599    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr" event={"ID":"a61c2cbb-e090-4ca1-825b-e0b85151fcf2","Type":"ContainerStarted","Data":"5e45cf6ebdb6cc8da423503b3db763b567f78d127df3673ba496063882d46e19"}
Jul 14 14:30:25.890506 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.890429    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dh9dt" event={"ID":"9e7ba746-3e33-4f52-bd37-1e2d9242b207","Type":"ContainerStarted","Data":"ef7d5508230342ae8425e9c304ef420a1cc2025dbabf66d08180013f35f1cb25"}
Jul 14 14:30:25.893556 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.893532    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal" event={"ID":"0a154078233a1a53525b6fe59aa34665","Type":"ContainerStarted","Data":"c5761f541c40d69ab75252f89e40cab08da863ed0015a7f47f36acbb223e273a"}
Jul 14 14:30:25.896591 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.896563    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nqtwh" event={"ID":"74295487-4d63-42cb-b367-fda447a5217b","Type":"ContainerStarted","Data":"cab4966aace5341901eb4903778c83993d43cc6238a8ec20ffc893548395a957"}
Jul 14 14:30:25.903779 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.903748    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fl6rb" event={"ID":"401faa4d-42da-4c1c-af1c-fe150089dfe0","Type":"ContainerStarted","Data":"ee64455f421f3a0066a2d5e939cf627c6ffa35610c9d983ded73a291eb7fc0a8"}
Jul 14 14:30:25.908705 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.908670    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kkp8m" event={"ID":"f1cc50dd-47f7-4a20-8047-0cef69221ee1","Type":"ContainerStarted","Data":"d8e2746ad8a1235b72b421aa1a26e624284268ebfd934039e1cf34bc6c6dad84"}
Jul 14 14:30:25.916710 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.916544    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65" event={"ID":"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55","Type":"ContainerStarted","Data":"6290c7296b06489a7ba50035bae435b5257f3923a5ef2f34d7ddd78ddfd8e7e7"}
Jul 14 14:30:25.918717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.918440    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nwmln" event={"ID":"5a8692c5-50c8-4012-8415-6d580d7ca0bb","Type":"ContainerStarted","Data":"d0758389ba96cdfce4c9bf4370541e766e412744e760a4b3dab576a7baf691f7"}
Jul 14 14:30:25.925987 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:25.925958    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nj27g" event={"ID":"76945752-395d-4809-abb1-cd8e68793179","Type":"ContainerStarted","Data":"ebec0ad070a690ab281c3901471b41c8fedd6edca20f8b50dcf94cb12b25614d"}
Jul 14 14:30:26.382245 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:26.382204    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gmw\" (UniqueName: \"kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw\") pod \"network-check-target-84g4n\" (UID: \"667c8f62-f765-4395-9740-d0c0bc87102c\") " pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:26.382509 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:26.382434    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 14:30:26.382509 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:26.382472    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 14:30:26.382509 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:26.382485    2586 projected.go:196] Error preparing data for projected volume kube-api-access-d4gmw for pod openshift-network-diagnostics/network-check-target-84g4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 14:30:26.382768 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:26.382553    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw podName:667c8f62-f765-4395-9740-d0c0bc87102c nodeName:}" failed. No retries permitted until 2026-07-14 14:30:28.382532744 +0000 UTC m=+6.048719145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4gmw" (UniqueName: "kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw") pod "network-check-target-84g4n" (UID: "667c8f62-f765-4395-9740-d0c0bc87102c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 14:30:26.400220 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:26.400189    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 14:30:26.483081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:26.483042    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:26.483249 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:26.483120    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs\") pod \"network-metrics-daemon-zqd7b\" (UID: \"85b30a97-2ff1-4091-9e77-4359a1c4c242\") " pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:26.483309 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:26.483279    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 14:30:26.483378 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:26.483349    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs podName:85b30a97-2ff1-4091-9e77-4359a1c4c242 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:28.483328471 +0000 UTC m=+6.149514875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs") pod "network-metrics-daemon-zqd7b" (UID: "85b30a97-2ff1-4091-9e77-4359a1c4c242") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 14:30:26.483798 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:26.483779    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 14:30:26.483898 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:26.483835    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret podName:3febdf7d-4462-4194-a512-7c860fc44797 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:28.483818647 +0000 UTC m=+6.150005056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret") pod "global-pull-secret-syncer-rlbbv" (UID: "3febdf7d-4462-4194-a512-7c860fc44797") : object "kube-system"/"original-pull-secret" not registered
Jul 14 14:30:26.799169 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:26.799113    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 14:25:25 +0000 UTC" deadline="2028-02-14 06:11:24.236480847 +0000 UTC"
Jul 14 14:30:26.799748 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:26.799202    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13911h40m57.437326943s"
Jul 14 14:30:26.871646 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:26.871612    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:26.871843 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:26.871753    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:26.872183 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:26.872156    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:26.872321 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:26.872255    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:26.872525 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:26.872467    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:26.872637 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:26.872561    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:28.397661 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:28.397619    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gmw\" (UniqueName: \"kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw\") pod \"network-check-target-84g4n\" (UID: \"667c8f62-f765-4395-9740-d0c0bc87102c\") " pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:28.398149 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:28.397825    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 14:30:28.398149 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:28.397850    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 14:30:28.398149 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:28.397864    2586 projected.go:196] Error preparing data for projected volume kube-api-access-d4gmw for pod openshift-network-diagnostics/network-check-target-84g4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 14:30:28.398149 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:28.397926    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw podName:667c8f62-f765-4395-9740-d0c0bc87102c nodeName:}" failed. No retries permitted until 2026-07-14 14:30:32.397907127 +0000 UTC m=+10.064093532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4gmw" (UniqueName: "kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw") pod "network-check-target-84g4n" (UID: "667c8f62-f765-4395-9740-d0c0bc87102c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 14:30:28.499735 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:28.499040    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs\") pod \"network-metrics-daemon-zqd7b\" (UID: \"85b30a97-2ff1-4091-9e77-4359a1c4c242\") " pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:28.499735 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:28.499146    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:28.499735 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:28.499282    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 14:30:28.499735 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:28.499351    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret podName:3febdf7d-4462-4194-a512-7c860fc44797 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:32.499331399 +0000 UTC m=+10.165517825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret") pod "global-pull-secret-syncer-rlbbv" (UID: "3febdf7d-4462-4194-a512-7c860fc44797") : object "kube-system"/"original-pull-secret" not registered
Jul 14 14:30:28.500084 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:28.499796    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 14:30:28.500084 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:28.499851    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs podName:85b30a97-2ff1-4091-9e77-4359a1c4c242 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:32.499834023 +0000 UTC m=+10.166020429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs") pod "network-metrics-daemon-zqd7b" (UID: "85b30a97-2ff1-4091-9e77-4359a1c4c242") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 14:30:28.871687 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:28.871654    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:28.871926 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:28.871696    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:28.871926 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:28.871654    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:28.871926 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:28.871819    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:28.872097 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:28.871926    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:28.872097 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:28.872009    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:30.871115 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:30.870570    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:30.871115 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:30.870583    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:30.871115 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:30.870712    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:30.871115 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:30.870785    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:30.871115 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:30.870585    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:30.871115 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:30.870893    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:32.434365 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:32.434322    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gmw\" (UniqueName: \"kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw\") pod \"network-check-target-84g4n\" (UID: \"667c8f62-f765-4395-9740-d0c0bc87102c\") " pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:32.434861 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:32.434550    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 14:30:32.434861 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:32.434584    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 14:30:32.434861 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:32.434599    2586 projected.go:196] Error preparing data for projected volume kube-api-access-d4gmw for pod openshift-network-diagnostics/network-check-target-84g4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 14:30:32.434861 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:32.434667    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw podName:667c8f62-f765-4395-9740-d0c0bc87102c nodeName:}" failed. No retries permitted until 2026-07-14 14:30:40.434647584 +0000 UTC m=+18.100834001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4gmw" (UniqueName: "kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw") pod "network-check-target-84g4n" (UID: "667c8f62-f765-4395-9740-d0c0bc87102c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 14:30:32.535722 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:32.535685    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs\") pod \"network-metrics-daemon-zqd7b\" (UID: \"85b30a97-2ff1-4091-9e77-4359a1c4c242\") " pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:32.535938 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:32.535764    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:32.535938 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:32.535914    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 14:30:32.536065 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:32.535980    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret podName:3febdf7d-4462-4194-a512-7c860fc44797 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:40.535960594 +0000 UTC m=+18.202146998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret") pod "global-pull-secret-syncer-rlbbv" (UID: "3febdf7d-4462-4194-a512-7c860fc44797") : object "kube-system"/"original-pull-secret" not registered
Jul 14 14:30:32.536065 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:32.536052    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 14:30:32.536183 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:32.536089    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs podName:85b30a97-2ff1-4091-9e77-4359a1c4c242 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:40.536076831 +0000 UTC m=+18.202263231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs") pod "network-metrics-daemon-zqd7b" (UID: "85b30a97-2ff1-4091-9e77-4359a1c4c242") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 14:30:32.870894 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:32.870809    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:32.871054 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:32.870949    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:32.871054 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:32.870979    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:32.871177 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:32.871089    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:32.871238 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:32.871223    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:32.871335 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:32.871314    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:34.871275 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:34.870576    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:34.871275 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:34.870716    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:34.871275 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:34.871152    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:34.871275 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:34.871224    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:34.871973 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:34.871413    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:34.871973 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:34.871557    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:36.870446 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:36.870382    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:36.870446 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:36.870429    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:36.870929 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:36.870827    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:36.870929 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:36.870804    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:36.871117 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:36.871090    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:36.871283 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:36.871251    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:38.870625 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:38.870588    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:38.871017 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:38.870601    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:38.871017 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:38.870708    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:38.871017 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:38.870722    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:38.871017 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:38.870824    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:38.871017 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:38.870906    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:40.502869 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:40.502829    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gmw\" (UniqueName: \"kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw\") pod \"network-check-target-84g4n\" (UID: \"667c8f62-f765-4395-9740-d0c0bc87102c\") " pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:40.503408 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:40.503042    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 14:30:40.503408 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:40.503070    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 14:30:40.503408 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:40.503088    2586 projected.go:196] Error preparing data for projected volume kube-api-access-d4gmw for pod openshift-network-diagnostics/network-check-target-84g4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 14:30:40.503408 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:40.503155    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw podName:667c8f62-f765-4395-9740-d0c0bc87102c nodeName:}" failed. No retries permitted until 2026-07-14 14:30:56.503135729 +0000 UTC m=+34.169322128 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4gmw" (UniqueName: "kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw") pod "network-check-target-84g4n" (UID: "667c8f62-f765-4395-9740-d0c0bc87102c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 14:30:40.604007 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:40.603975    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:40.604203 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:40.604034    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs\") pod \"network-metrics-daemon-zqd7b\" (UID: \"85b30a97-2ff1-4091-9e77-4359a1c4c242\") " pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:40.604203 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:40.604150    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 14:30:40.604203 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:40.604150    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 14:30:40.604342 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:40.604219    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs podName:85b30a97-2ff1-4091-9e77-4359a1c4c242 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:56.604198628 +0000 UTC m=+34.270385030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs") pod "network-metrics-daemon-zqd7b" (UID: "85b30a97-2ff1-4091-9e77-4359a1c4c242") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 14:30:40.604342 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:40.604241    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret podName:3febdf7d-4462-4194-a512-7c860fc44797 nodeName:}" failed. No retries permitted until 2026-07-14 14:30:56.604229969 +0000 UTC m=+34.270416374 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret") pod "global-pull-secret-syncer-rlbbv" (UID: "3febdf7d-4462-4194-a512-7c860fc44797") : object "kube-system"/"original-pull-secret" not registered
Jul 14 14:30:40.870391 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:40.870303    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:40.870578 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:40.870299    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:40.870578 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:40.870459    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:40.870578 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:40.870460    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:40.870578 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:40.870538    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:40.870798 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:40.870601    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:42.871595 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:42.871558    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:42.872050 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:42.871692    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:42.872050 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:42.871678    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:42.872050 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:42.871812    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:42.872050 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:42.871862    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:42.872050 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:42.871942    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:44.872093 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:44.872061    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:44.872592 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:44.872188    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:44.872664 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:44.872606    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:44.872772 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:44.872713    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:44.872840 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:44.872776    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:44.872895 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:44.872842    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:46.871255 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:46.870995    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:46.871890 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:46.871239    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:46.871890 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:46.871378    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:46.871890 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:46.871646    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:46.871890 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:46.871726    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:46.871890 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:46.871814    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:46.984696 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:46.984668    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:30:46.985099 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:46.985074    2586 generic.go:356] "Generic (PLEG): container finished" podID="a61c2cbb-e090-4ca1-825b-e0b85151fcf2" containerID="875bf11473902cb210065a2f9757e265894113b0bf58bdcea5dba21ff6da22f2" exitCode=1
Jul 14 14:30:46.985218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:46.985170    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr" event={"ID":"a61c2cbb-e090-4ca1-825b-e0b85151fcf2","Type":"ContainerStarted","Data":"96ca7a74b9668e57426408f7d1a5248212ba265ca9a459cbc1e89d05a294149e"}
Jul 14 14:30:46.985303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:46.985283    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr" event={"ID":"a61c2cbb-e090-4ca1-825b-e0b85151fcf2","Type":"ContainerDied","Data":"875bf11473902cb210065a2f9757e265894113b0bf58bdcea5dba21ff6da22f2"}
Jul 14 14:30:46.985354 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:46.985344    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr" event={"ID":"a61c2cbb-e090-4ca1-825b-e0b85151fcf2","Type":"ContainerStarted","Data":"82cedb0c099c08401d409c210a363ec50771cdea6c184223ef0497437257caa0"}
Jul 14 14:30:46.986716 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:46.986695    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dh9dt" event={"ID":"9e7ba746-3e33-4f52-bd37-1e2d9242b207","Type":"ContainerStarted","Data":"4a2f702a6492e6d4d790bd45a8493bf879f7e3498f9de355d2ae976ef2bd264b"}
Jul 14 14:30:46.989479 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:46.989453    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-209.ec2.internal" event={"ID":"32e6ac82d25811564f3e9e457bd986bc","Type":"ContainerStarted","Data":"10a9c2189e723a564a9cdcea37272fa875c2ad4de9659f4027a6e271d1352f9f"}
Jul 14 14:30:46.993672 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:46.993642    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kkp8m" event={"ID":"f1cc50dd-47f7-4a20-8047-0cef69221ee1","Type":"ContainerStarted","Data":"6806e5a55ac4d95b55160b80b58310cc04cb46c8235f700ae55d45591e99c5ef"}
Jul 14 14:30:47.030217 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:47.028972    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-209.ec2.internal" podStartSLOduration=22.028928082 podStartE2EDuration="22.028928082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:47.028562027 +0000 UTC m=+24.694748448" watchObservedRunningTime="2026-07-14 14:30:47.028928082 +0000 UTC m=+24.695114503"
Jul 14 14:30:47.030217 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:47.029504    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dh9dt" podStartSLOduration=2.528366149 podStartE2EDuration="24.029488274s" totalImagesPullingTime="21.501122125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:47.008562267 +0000 UTC m=+24.674748690" watchObservedRunningTime="2026-07-14 14:30:47.029488274 +0000 UTC m=+24.695674696"
Jul 14 14:30:47.052887 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:47.052729    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kkp8m" podStartSLOduration=2.488000605 podStartE2EDuration="24.052715204s" totalImagesPullingTime="21.564714599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:47.05211966 +0000 UTC m=+24.718306080" watchObservedRunningTime="2026-07-14 14:30:47.052715204 +0000 UTC m=+24.718901624"
Jul 14 14:30:47.998014 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:47.997772    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nqtwh" event={"ID":"74295487-4d63-42cb-b367-fda447a5217b","Type":"ContainerStarted","Data":"df84343bd4dc2b4a49dbbae9b8c629c696139dc417230f80d65f964f84035428"}
Jul 14 14:30:47.999724 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:47.999575    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fl6rb" event={"ID":"401faa4d-42da-4c1c-af1c-fe150089dfe0","Type":"ContainerStarted","Data":"8c8a23399b46bdebda62b89ad6300f45841f6e76543327ecf604c32879a0e570"}
Jul 14 14:30:48.001155 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.001127    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65" event={"ID":"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55","Type":"ContainerStarted","Data":"7663e9b8fd41e1429ac96adb10ea59c22bdbf4579071e5a0a0eabf0e76e7ffba"}
Jul 14 14:30:48.002653 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.002625    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nwmln" event={"ID":"5a8692c5-50c8-4012-8415-6d580d7ca0bb","Type":"ContainerStarted","Data":"7ced1eb7b521030798fe5986db1be5303c4b5b3f568c19f6ae010f2334c35f2c"}
Jul 14 14:30:48.004286 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.004259    2586 generic.go:356] "Generic (PLEG): container finished" podID="76945752-395d-4809-abb1-cd8e68793179" containerID="751b0d0f520b8af993c601e56344bf9c3cff341ff9a452f4a1d23499e37a23db" exitCode=0
Jul 14 14:30:48.004391 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.004336    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nj27g" event={"ID":"76945752-395d-4809-abb1-cd8e68793179","Type":"ContainerDied","Data":"751b0d0f520b8af993c601e56344bf9c3cff341ff9a452f4a1d23499e37a23db"}
Jul 14 14:30:48.006395 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.006366    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t84p9" event={"ID":"f12e6b3b-d351-4faf-bf16-9ccfd4100848","Type":"ContainerStarted","Data":"1d847529100361a65d8a9b23b91d6eb2f1c6feec9c037081f4ef7e1696c3a5d9"}
Jul 14 14:30:48.009368 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.009353    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:30:48.009769 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.009734    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr" event={"ID":"a61c2cbb-e090-4ca1-825b-e0b85151fcf2","Type":"ContainerStarted","Data":"69d3ea492e5bc41cb959f95e6acaa8ffd9e2c66278a74550d7e18de242c15173"}
Jul 14 14:30:48.009838 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.009807    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr" event={"ID":"a61c2cbb-e090-4ca1-825b-e0b85151fcf2","Type":"ContainerStarted","Data":"33ef5a3720bdc1e97ee25a09b7815a88256bf9a12efca1e6a007be62d39e9ee7"}
Jul 14 14:30:48.009884 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.009857    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr" event={"ID":"a61c2cbb-e090-4ca1-825b-e0b85151fcf2","Type":"ContainerStarted","Data":"2d648800d42075de2ad82b2a5a3f4c50151dcbaa7f8a5ca78a560acc901bb9e6"}
Jul 14 14:30:48.011380 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.011355    2586 generic.go:356] "Generic (PLEG): container finished" podID="0a154078233a1a53525b6fe59aa34665" containerID="29ecd5bdbe1afe1bb16b6dbaf14afbae1375e38059d61a7975660f0015a363bb" exitCode=0
Jul 14 14:30:48.011479 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.011426    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal" event={"ID":"0a154078233a1a53525b6fe59aa34665","Type":"ContainerDied","Data":"29ecd5bdbe1afe1bb16b6dbaf14afbae1375e38059d61a7975660f0015a363bb"}
Jul 14 14:30:48.044618 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.044557    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nqtwh" podStartSLOduration=2.5594132590000003 podStartE2EDuration="24.044519573s" totalImagesPullingTime="21.485106314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:48.020103623 +0000 UTC m=+25.686290077" watchObservedRunningTime="2026-07-14 14:30:48.044519573 +0000 UTC m=+25.710705997"
Jul 14 14:30:48.061802 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.061755    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nwmln" podStartSLOduration=3.580586668 podStartE2EDuration="25.061738557s" totalImagesPullingTime="21.481151889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:48.061542814 +0000 UTC m=+25.727729237" watchObservedRunningTime="2026-07-14 14:30:48.061738557 +0000 UTC m=+25.727924994"
Jul 14 14:30:48.080527 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.080475    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fl6rb" podStartSLOduration=3.586525362 podStartE2EDuration="25.080459514s" totalImagesPullingTime="21.493934152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:48.080152252 +0000 UTC m=+25.746338673" watchObservedRunningTime="2026-07-14 14:30:48.080459514 +0000 UTC m=+25.746645934"
Jul 14 14:30:48.099659 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.099601    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t84p9" podStartSLOduration=2.641905329 podStartE2EDuration="24.099582124s" totalImagesPullingTime="21.457676795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:48.099345182 +0000 UTC m=+25.765531604" watchObservedRunningTime="2026-07-14 14:30:48.099582124 +0000 UTC m=+25.765768545"
Jul 14 14:30:48.340323 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.340290    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t84p9"
Jul 14 14:30:48.340951 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.340934    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t84p9"
Jul 14 14:30:48.574913 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.574875    2586 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 14 14:30:48.831033 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.830664    2586 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T14:30:48.574907847Z","UUID":"6394d52e-e47a-466a-b9b3-d37b67107d8d","Handler":null,"Name":"","Endpoint":""}
Jul 14 14:30:48.832233 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.832207    2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 14 14:30:48.832383 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.832251    2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 14 14:30:48.870731 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.870698    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:48.870731 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.870710    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:48.870984 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:48.870833    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:48.870984 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:48.870911    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:48.871096 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:48.871003    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:48.871096 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:48.871085    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:49.015987 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:49.015948    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65" event={"ID":"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55","Type":"ContainerStarted","Data":"53af71f309fca11946cd004984c61a39fdc91ceee7cdb6ff5bb8a6d5e2c5641a"}
Jul 14 14:30:49.018056 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:49.017972    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal" event={"ID":"0a154078233a1a53525b6fe59aa34665","Type":"ContainerStarted","Data":"f6208a31793ec00603cc735e3b1c4fb8c54d4bc6bc8bc172e6bb6d9c1a81d73e"}
Jul 14 14:30:49.019081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:49.018846    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t84p9"
Jul 14 14:30:49.019264 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:49.019243    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t84p9"
Jul 14 14:30:49.045476 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:49.045403    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-209.ec2.internal" podStartSLOduration=24.045382272 podStartE2EDuration="24.045382272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:49.044575719 +0000 UTC m=+26.710762139" watchObservedRunningTime="2026-07-14 14:30:49.045382272 +0000 UTC m=+26.711568694"
Jul 14 14:30:50.023848 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:50.023808    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:30:50.024405 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:50.024272    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr" event={"ID":"a61c2cbb-e090-4ca1-825b-e0b85151fcf2","Type":"ContainerStarted","Data":"e143d503cd7ee3d7cc4bf7ca4a1b8ed9a44512d0a21097d82850a137bc8aafdf"}
Jul 14 14:30:50.026536 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:50.026509    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65" event={"ID":"c91bb9dd-9ad3-4d3d-8555-b5f61c681f55","Type":"ContainerStarted","Data":"01c3e0a5c9c7cc896a6b086ceac24b541ab180ea0955e0fcadd62b957340a275"}
Jul 14 14:30:50.046341 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:50.046285    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4b65" podStartSLOduration=2.987291815 podStartE2EDuration="27.046267765s" totalImagesPullingTime="24.05897595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:50.04595397 +0000 UTC m=+27.712140388" watchObservedRunningTime="2026-07-14 14:30:50.046267765 +0000 UTC m=+27.712454185"
Jul 14 14:30:50.870216 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:50.870178    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:50.870401 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:50.870178    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:50.870401 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:50.870323    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:50.870401 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:50.870178    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:50.870587 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:50.870402    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:50.870587 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:50.870468    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:52.872345 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:52.872045    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:52.872986 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:52.872099    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:52.872986 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:52.872397    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:52.872986 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:52.872125    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:52.872986 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:52.872454    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:52.872986 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:52.872534    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:53.036623 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:53.036598    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:30:53.037006 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:53.036972    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr" event={"ID":"a61c2cbb-e090-4ca1-825b-e0b85151fcf2","Type":"ContainerStarted","Data":"8449eb4dad1032ba330c6eeb4710b82bcf93b17c1620315266a25223af12466e"}
Jul 14 14:30:53.037318 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:53.037298    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:53.037499 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:53.037482    2586 scope.go:117] "RemoveContainer" containerID="875bf11473902cb210065a2f9757e265894113b0bf58bdcea5dba21ff6da22f2"
Jul 14 14:30:53.039105 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:53.039084    2586 generic.go:356] "Generic (PLEG): container finished" podID="76945752-395d-4809-abb1-cd8e68793179" containerID="ecb52b4c29e8588810e5bdb04a3ee890355fcf79b5a1815a8ce3a94a46be302d" exitCode=0
Jul 14 14:30:53.039188 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:53.039116    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nj27g" event={"ID":"76945752-395d-4809-abb1-cd8e68793179","Type":"ContainerDied","Data":"ecb52b4c29e8588810e5bdb04a3ee890355fcf79b5a1815a8ce3a94a46be302d"}
Jul 14 14:30:53.055324 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:53.055302    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:54.044800 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:54.044767    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:30:54.045402 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:54.045061    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr" event={"ID":"a61c2cbb-e090-4ca1-825b-e0b85151fcf2","Type":"ContainerStarted","Data":"2150e3f7b157da95bcc4deea8ad6248e5a146b612b0a830d9df91af8a0298c8a"}
Jul 14 14:30:54.045402 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:54.045306    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:54.045402 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:54.045380    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:54.061237 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:54.061210    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:30:54.075512 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:54.075467    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr" podStartSLOduration=9.488245789 podStartE2EDuration="31.075453037s" totalImagesPullingTime="21.587207248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:30:54.074042276 +0000 UTC m=+31.740228700" watchObservedRunningTime="2026-07-14 14:30:54.075453037 +0000 UTC m=+31.741639457"
Jul 14 14:30:54.453212 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:54.453040    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zqd7b"]
Jul 14 14:30:54.453365 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:54.453316    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:54.453444 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:54.453406    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:54.453565 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:54.453545    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rlbbv"]
Jul 14 14:30:54.453645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:54.453635    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:54.453724 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:54.453708    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:54.455098 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:54.455074    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-84g4n"]
Jul 14 14:30:54.455218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:54.455163    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:54.455274 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:54.455233    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:55.049616 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:55.049586    2586 generic.go:356] "Generic (PLEG): container finished" podID="76945752-395d-4809-abb1-cd8e68793179" containerID="eddcb297d6a4456bb3d41eb33539537cd9fb3f845fc1fb2a57d7355041c1c576" exitCode=0
Jul 14 14:30:55.050242 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:55.049661    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nj27g" event={"ID":"76945752-395d-4809-abb1-cd8e68793179","Type":"ContainerDied","Data":"eddcb297d6a4456bb3d41eb33539537cd9fb3f845fc1fb2a57d7355041c1c576"}
Jul 14 14:30:55.870184 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:55.870156    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:55.870184 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:55.870182    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:55.870363 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:55.870156    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:55.870363 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:55.870250    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:55.870363 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:55.870334    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:55.870478 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:55.870399    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:56.524865 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:56.524835    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gmw\" (UniqueName: \"kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw\") pod \"network-check-target-84g4n\" (UID: \"667c8f62-f765-4395-9740-d0c0bc87102c\") " pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:56.525359 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:56.524975    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 14:30:56.525359 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:56.524991    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 14:30:56.525359 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:56.525000    2586 projected.go:196] Error preparing data for projected volume kube-api-access-d4gmw for pod openshift-network-diagnostics/network-check-target-84g4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 14:30:56.525359 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:56.525050    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw podName:667c8f62-f765-4395-9740-d0c0bc87102c nodeName:}" failed. No retries permitted until 2026-07-14 14:31:28.525034812 +0000 UTC m=+66.191221211 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4gmw" (UniqueName: "kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw") pod "network-check-target-84g4n" (UID: "667c8f62-f765-4395-9740-d0c0bc87102c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 14:30:56.625396 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:56.625364    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs\") pod \"network-metrics-daemon-zqd7b\" (UID: \"85b30a97-2ff1-4091-9e77-4359a1c4c242\") " pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:56.625563 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:56.625442    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:56.625563 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:56.625511    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 14:30:56.625563 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:56.625527    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 14:30:56.625657 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:56.625570    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret podName:3febdf7d-4462-4194-a512-7c860fc44797 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:28.625558444 +0000 UTC m=+66.291744843 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret") pod "global-pull-secret-syncer-rlbbv" (UID: "3febdf7d-4462-4194-a512-7c860fc44797") : object "kube-system"/"original-pull-secret" not registered
Jul 14 14:30:56.625657 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:56.625587    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs podName:85b30a97-2ff1-4091-9e77-4359a1c4c242 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:28.625578268 +0000 UTC m=+66.291764667 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs") pod "network-metrics-daemon-zqd7b" (UID: "85b30a97-2ff1-4091-9e77-4359a1c4c242") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 14:30:57.057380 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:57.057350    2586 generic.go:356] "Generic (PLEG): container finished" podID="76945752-395d-4809-abb1-cd8e68793179" containerID="642fea9515bd4e406d1b038ba0ec447d018848a937766f7aba3ab4d23665e058" exitCode=0
Jul 14 14:30:57.057550 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:57.057402    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nj27g" event={"ID":"76945752-395d-4809-abb1-cd8e68793179","Type":"ContainerDied","Data":"642fea9515bd4e406d1b038ba0ec447d018848a937766f7aba3ab4d23665e058"}
Jul 14 14:30:57.870585 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:57.870551    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:57.870585 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:57.870576    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:57.870585 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:57.870586    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:57.871174 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:57.870696    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqd7b" podUID="85b30a97-2ff1-4091-9e77-4359a1c4c242"
Jul 14 14:30:57.871174 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:57.870858    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rlbbv" podUID="3febdf7d-4462-4194-a512-7c860fc44797"
Jul 14 14:30:57.871174 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:57.870990    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-84g4n" podUID="667c8f62-f765-4395-9740-d0c0bc87102c"
Jul 14 14:30:59.618649 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.618575    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-209.ec2.internal" event="NodeReady"
Jul 14 14:30:59.619048 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.618708    2586 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 14 14:30:59.664888 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.664856    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2f5v2"]
Jul 14 14:30:59.667515 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.667489    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:30:59.667963 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.667938    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wxlrn"]
Jul 14 14:30:59.669580 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.669562    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wxlrn"
Jul 14 14:30:59.670153 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.670133    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 14 14:30:59.670244 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.670188    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x4st5\""
Jul 14 14:30:59.670296 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.670140    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 14 14:30:59.672037 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.671972    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 14 14:30:59.672164 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.672148    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fdbdg\""
Jul 14 14:30:59.672225 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.672154    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 14 14:30:59.672635 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.672614    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 14 14:30:59.679577 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.679556    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2f5v2"]
Jul 14 14:30:59.681966 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.681938    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wxlrn"]
Jul 14 14:30:59.749573 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.749538    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwlqc\" (UniqueName: \"kubernetes.io/projected/8b115d5b-3bdb-4b02-b747-a749b79536e4-kube-api-access-mwlqc\") pod \"ingress-canary-wxlrn\" (UID: \"8b115d5b-3bdb-4b02-b747-a749b79536e4\") " pod="openshift-ingress-canary/ingress-canary-wxlrn"
Jul 14 14:30:59.749761 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.749584    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:30:59.749761 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.749624    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd676278-6891-49d4-881e-41f5b8029e62-config-volume\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:30:59.749761 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.749688    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b115d5b-3bdb-4b02-b747-a749b79536e4-cert\") pod \"ingress-canary-wxlrn\" (UID: \"8b115d5b-3bdb-4b02-b747-a749b79536e4\") " pod="openshift-ingress-canary/ingress-canary-wxlrn"
Jul 14 14:30:59.749761 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.749724    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l88ld\" (UniqueName: \"kubernetes.io/projected/dd676278-6891-49d4-881e-41f5b8029e62-kube-api-access-l88ld\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:30:59.749976 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.749765    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd676278-6891-49d4-881e-41f5b8029e62-tmp-dir\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:30:59.850623 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.850581    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd676278-6891-49d4-881e-41f5b8029e62-config-volume\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:30:59.850809 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.850642    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b115d5b-3bdb-4b02-b747-a749b79536e4-cert\") pod \"ingress-canary-wxlrn\" (UID: \"8b115d5b-3bdb-4b02-b747-a749b79536e4\") " pod="openshift-ingress-canary/ingress-canary-wxlrn"
Jul 14 14:30:59.850809 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.850676    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l88ld\" (UniqueName: \"kubernetes.io/projected/dd676278-6891-49d4-881e-41f5b8029e62-kube-api-access-l88ld\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:30:59.850809 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.850707    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd676278-6891-49d4-881e-41f5b8029e62-tmp-dir\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:30:59.850809 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.850769    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwlqc\" (UniqueName: \"kubernetes.io/projected/8b115d5b-3bdb-4b02-b747-a749b79536e4-kube-api-access-mwlqc\") pod \"ingress-canary-wxlrn\" (UID: \"8b115d5b-3bdb-4b02-b747-a749b79536e4\") " pod="openshift-ingress-canary/ingress-canary-wxlrn"
Jul 14 14:30:59.850809 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.850807    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:30:59.851052 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:59.850902    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 14:30:59.851052 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:30:59.850957    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls podName:dd676278-6891-49d4-881e-41f5b8029e62 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:00.350938889 +0000 UTC m=+38.017125309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls") pod "dns-default-2f5v2" (UID: "dd676278-6891-49d4-881e-41f5b8029e62") : secret "dns-default-metrics-tls" not found
Jul 14 14:30:59.851315 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.851291    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd676278-6891-49d4-881e-41f5b8029e62-config-volume\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:30:59.851589 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.851567    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd676278-6891-49d4-881e-41f5b8029e62-tmp-dir\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:30:59.855617 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.855473    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b115d5b-3bdb-4b02-b747-a749b79536e4-cert\") pod \"ingress-canary-wxlrn\" (UID: \"8b115d5b-3bdb-4b02-b747-a749b79536e4\") " pod="openshift-ingress-canary/ingress-canary-wxlrn"
Jul 14 14:30:59.860152 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.860124    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l88ld\" (UniqueName: \"kubernetes.io/projected/dd676278-6891-49d4-881e-41f5b8029e62-kube-api-access-l88ld\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:30:59.860258 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.860155    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwlqc\" (UniqueName: \"kubernetes.io/projected/8b115d5b-3bdb-4b02-b747-a749b79536e4-kube-api-access-mwlqc\") pod \"ingress-canary-wxlrn\" (UID: \"8b115d5b-3bdb-4b02-b747-a749b79536e4\") " pod="openshift-ingress-canary/ingress-canary-wxlrn"
Jul 14 14:30:59.870090 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.870014    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:30:59.870171 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.870113    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:30:59.870463 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.870397    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:30:59.873429 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.873393    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 14:30:59.873529 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.873440    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 14:30:59.873529 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.873393    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 14:30:59.873529 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.873488    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-shzjn\""
Jul 14 14:30:59.873690 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.873528    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lqsw5\""
Jul 14 14:30:59.873763 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.873748    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 14:30:59.987343 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:30:59.987306    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wxlrn"
Jul 14 14:31:00.354680 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:00.354641    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:31:00.354868 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:00.354816    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 14:31:00.354926 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:00.354882    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls podName:dd676278-6891-49d4-881e-41f5b8029e62 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:01.354865367 +0000 UTC m=+39.021051768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls") pod "dns-default-2f5v2" (UID: "dd676278-6891-49d4-881e-41f5b8029e62") : secret "dns-default-metrics-tls" not found
Jul 14 14:31:01.240670 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:01.240637    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wxlrn"]
Jul 14 14:31:01.245652 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:31:01.245626    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b115d5b_3bdb_4b02_b747_a749b79536e4.slice/crio-ae83ff31e18e44188667bd44937ad73ac06db3c24fdb61ff6c33876da407ed79 WatchSource:0}: Error finding container ae83ff31e18e44188667bd44937ad73ac06db3c24fdb61ff6c33876da407ed79: Status 404 returned error can't find the container with id ae83ff31e18e44188667bd44937ad73ac06db3c24fdb61ff6c33876da407ed79
Jul 14 14:31:01.362738 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:01.362677    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:31:01.362861 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:01.362842    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 14:31:01.362927 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:01.362917    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls podName:dd676278-6891-49d4-881e-41f5b8029e62 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:03.36289612 +0000 UTC m=+41.029082536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls") pod "dns-default-2f5v2" (UID: "dd676278-6891-49d4-881e-41f5b8029e62") : secret "dns-default-metrics-tls" not found
Jul 14 14:31:02.072512 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:02.072479    2586 generic.go:356] "Generic (PLEG): container finished" podID="76945752-395d-4809-abb1-cd8e68793179" containerID="7cb6640aaf543dba27fddf3c001e8cba64abc274ff0461bcb6d5e4be56855966" exitCode=0
Jul 14 14:31:02.072704 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:02.072567    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nj27g" event={"ID":"76945752-395d-4809-abb1-cd8e68793179","Type":"ContainerDied","Data":"7cb6640aaf543dba27fddf3c001e8cba64abc274ff0461bcb6d5e4be56855966"}
Jul 14 14:31:02.073692 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:02.073672    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wxlrn" event={"ID":"8b115d5b-3bdb-4b02-b747-a749b79536e4","Type":"ContainerStarted","Data":"ae83ff31e18e44188667bd44937ad73ac06db3c24fdb61ff6c33876da407ed79"}
Jul 14 14:31:03.080715 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:03.080514    2586 generic.go:356] "Generic (PLEG): container finished" podID="76945752-395d-4809-abb1-cd8e68793179" containerID="1977d74afe65db2d522bfec711918b6e970ba0491ab3ca3d08ae18f0c0976340" exitCode=0
Jul 14 14:31:03.081113 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:03.080855    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nj27g" event={"ID":"76945752-395d-4809-abb1-cd8e68793179","Type":"ContainerDied","Data":"1977d74afe65db2d522bfec711918b6e970ba0491ab3ca3d08ae18f0c0976340"}
Jul 14 14:31:03.378989 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:03.378895    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:31:03.379155 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:03.379090    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 14:31:03.379222 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:03.379174    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls podName:dd676278-6891-49d4-881e-41f5b8029e62 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:07.379152039 +0000 UTC m=+45.045338444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls") pod "dns-default-2f5v2" (UID: "dd676278-6891-49d4-881e-41f5b8029e62") : secret "dns-default-metrics-tls" not found
Jul 14 14:31:04.086312 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:04.086229    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nj27g" event={"ID":"76945752-395d-4809-abb1-cd8e68793179","Type":"ContainerStarted","Data":"46a71682c0d95d2e3c3be055f8a60c34d97e3a25a2bd22f7c2150eed0d47b9db"}
Jul 14 14:31:04.110668 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:04.110609    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nj27g" podStartSLOduration=8.513673999 podStartE2EDuration="41.110585899s" totalImagesPullingTime="32.5969119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:31:04.110014172 +0000 UTC m=+41.776200593" watchObservedRunningTime="2026-07-14 14:31:04.110585899 +0000 UTC m=+41.776772320"
Jul 14 14:31:05.090563 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:05.090523    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wxlrn" event={"ID":"8b115d5b-3bdb-4b02-b747-a749b79536e4","Type":"ContainerStarted","Data":"0f1a62851c2e03a46c5f55221cf83f257251b61922922481cfde07d08f0a68c9"}
Jul 14 14:31:07.406781 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:07.406744    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:31:07.407148 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:07.406917    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 14:31:07.407148 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:07.407007    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls podName:dd676278-6891-49d4-881e-41f5b8029e62 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:15.406981123 +0000 UTC m=+53.073167523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls") pod "dns-default-2f5v2" (UID: "dd676278-6891-49d4-881e-41f5b8029e62") : secret "dns-default-metrics-tls" not found
Jul 14 14:31:15.460890 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:15.460852    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:31:15.461505 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:15.461045    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 14:31:15.461505 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:15.461132    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls podName:dd676278-6891-49d4-881e-41f5b8029e62 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:31.461109262 +0000 UTC m=+69.127295682 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls") pod "dns-default-2f5v2" (UID: "dd676278-6891-49d4-881e-41f5b8029e62") : secret "dns-default-metrics-tls" not found
Jul 14 14:31:26.076426 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:26.076375    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x8sbr"
Jul 14 14:31:26.103368 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:26.103272    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wxlrn" podStartSLOduration=24.203634575 podStartE2EDuration="27.10325653s" totalImagesPullingTime="2.899621955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:31:05.107063889 +0000 UTC m=+42.773250312" watchObservedRunningTime="2026-07-14 14:31:26.10325653 +0000 UTC m=+63.769442951"
Jul 14 14:31:28.552188 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:28.552133    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gmw\" (UniqueName: \"kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw\") pod \"network-check-target-84g4n\" (UID: \"667c8f62-f765-4395-9740-d0c0bc87102c\") " pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:31:28.555314 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:28.555296    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 14:31:28.566746 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:28.566722    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 14:31:28.577107 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:28.577068    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4gmw\" (UniqueName: \"kubernetes.io/projected/667c8f62-f765-4395-9740-d0c0bc87102c-kube-api-access-d4gmw\") pod \"network-check-target-84g4n\" (UID: \"667c8f62-f765-4395-9740-d0c0bc87102c\") " pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:31:28.653271 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:28.653229    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:31:28.653486 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:28.653292    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs\") pod \"network-metrics-daemon-zqd7b\" (UID: \"85b30a97-2ff1-4091-9e77-4359a1c4c242\") " pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:31:28.656247 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:28.656223    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 14:31:28.656298 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:28.656263    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 14:31:28.664521 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:28.664492    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 14 14:31:28.664592 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:28.664581    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs podName:85b30a97-2ff1-4091-9e77-4359a1c4c242 nodeName:}" failed. No retries permitted until 2026-07-14 14:32:32.664562824 +0000 UTC m=+130.330749222 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs") pod "network-metrics-daemon-zqd7b" (UID: "85b30a97-2ff1-4091-9e77-4359a1c4c242") : secret "metrics-daemon-secret" not found
Jul 14 14:31:28.666837 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:28.666813    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3febdf7d-4462-4194-a512-7c860fc44797-original-pull-secret\") pod \"global-pull-secret-syncer-rlbbv\" (UID: \"3febdf7d-4462-4194-a512-7c860fc44797\") " pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:31:28.688126 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:28.688081    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rlbbv"
Jul 14 14:31:28.696866 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:28.696829    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lqsw5\""
Jul 14 14:31:28.704297 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:28.704267    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:31:28.825843 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:28.825752    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rlbbv"]
Jul 14 14:31:28.829659 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:31:28.829626    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3febdf7d_4462_4194_a512_7c860fc44797.slice/crio-a68238738230ad55672223a5f2ad557e2527abf1f56f9ddcd64549dead2638cd WatchSource:0}: Error finding container a68238738230ad55672223a5f2ad557e2527abf1f56f9ddcd64549dead2638cd: Status 404 returned error can't find the container with id a68238738230ad55672223a5f2ad557e2527abf1f56f9ddcd64549dead2638cd
Jul 14 14:31:28.841101 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:28.841070    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-84g4n"]
Jul 14 14:31:28.843859 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:31:28.843827    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667c8f62_f765_4395_9740_d0c0bc87102c.slice/crio-c700172c74101c020cfb5161ade45f89934350d3e65af233839f180700b85642 WatchSource:0}: Error finding container c700172c74101c020cfb5161ade45f89934350d3e65af233839f180700b85642: Status 404 returned error can't find the container with id c700172c74101c020cfb5161ade45f89934350d3e65af233839f180700b85642
Jul 14 14:31:29.148570 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:29.148492    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-84g4n" event={"ID":"667c8f62-f765-4395-9740-d0c0bc87102c","Type":"ContainerStarted","Data":"c700172c74101c020cfb5161ade45f89934350d3e65af233839f180700b85642"}
Jul 14 14:31:29.152756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:29.152714    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rlbbv" event={"ID":"3febdf7d-4462-4194-a512-7c860fc44797","Type":"ContainerStarted","Data":"a68238738230ad55672223a5f2ad557e2527abf1f56f9ddcd64549dead2638cd"}
Jul 14 14:31:31.471139 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:31.471093    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:31:31.471649 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:31.471272    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 14:31:31.471649 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:31.471556    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls podName:dd676278-6891-49d4-881e-41f5b8029e62 nodeName:}" failed. No retries permitted until 2026-07-14 14:32:03.471536465 +0000 UTC m=+101.137722863 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls") pod "dns-default-2f5v2" (UID: "dd676278-6891-49d4-881e-41f5b8029e62") : secret "dns-default-metrics-tls" not found
Jul 14 14:31:34.337712 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.336848    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-hqvkg"]
Jul 14 14:31:34.353153 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.352940    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"]
Jul 14 14:31:34.353153 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.353057    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.356046 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.356012    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 14 14:31:34.356176 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.356161    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 14 14:31:34.356845 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.356404    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 14 14:31:34.356845 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.356466    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 14 14:31:34.356845 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.356698    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-56c7v\""
Jul 14 14:31:34.367714 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.367686    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 14 14:31:34.368065 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.368032    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-hqvkg"]
Jul 14 14:31:34.368169 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.368149    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:31:34.369090 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.369063    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"]
Jul 14 14:31:34.371587 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.371559    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 14 14:31:34.371872 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.371851    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 14 14:31:34.372179 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.372160    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 14 14:31:34.373367 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.373351    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 14 14:31:34.373463 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.373364    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-424gg\""
Jul 14 14:31:34.392178 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.392138    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:31:34.392356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.392213    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwtqn\" (UniqueName: \"kubernetes.io/projected/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-kube-api-access-jwtqn\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:31:34.392356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.392258    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.392356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.392294    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-serving-cert\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.392522 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.392360    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.392522 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.392483    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8s96\" (UniqueName: \"kubernetes.io/projected/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-kube-api-access-r8s96\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.392620 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.392522    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-snapshots\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.392620 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.392555    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:31:34.392620 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.392589    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-tmp\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.443588 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.443549    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5965b849d4-jl74t"]
Jul 14 14:31:34.459837 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.459801    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5965b849d4-jl74t"]
Jul 14 14:31:34.460000 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.459928    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:34.462537 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.462511    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 14 14:31:34.462691 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.462611    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 14:31:34.462770 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.462755    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jwlpv\""
Jul 14 14:31:34.463599 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.463034    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 14 14:31:34.463599 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.463042    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 14:31:34.463599 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.463075    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 14 14:31:34.463599 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.463081    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 14 14:31:34.493164 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.493117    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.493164 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.493163    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-serving-cert\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.493380 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.493190    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.493380 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.493221    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-default-certificate\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:34.493380 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.493274    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8s96\" (UniqueName: \"kubernetes.io/projected/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-kube-api-access-r8s96\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.493380 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.493302    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-snapshots\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.493380 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.493327    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:31:34.493380 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.493358    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:34.495194 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.493385    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-tmp\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.495194 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.493427    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:34.495194 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.493455    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lsbg\" (UniqueName: \"kubernetes.io/projected/8bbbac86-f54f-4df4-9839-ecbc28741a9f-kube-api-access-2lsbg\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:34.495194 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.493530    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:31:34.495194 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.493553    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-stats-auth\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:34.495194 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.493591    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtqn\" (UniqueName: \"kubernetes.io/projected/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-kube-api-access-jwtqn\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:31:34.495194 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:34.494655    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 14 14:31:34.495194 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:34.494740    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls podName:a26ee4e3-b7fb-4ed4-9056-ae52a53b8667 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:34.99471687 +0000 UTC m=+72.660903286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-sgrwz" (UID: "a26ee4e3-b7fb-4ed4-9056-ae52a53b8667") : secret "cluster-monitoring-operator-tls" not found
Jul 14 14:31:34.495194 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.494847    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-tmp\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.495665 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.495523    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:31:34.495928 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.495906    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-snapshots\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.496380 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.496355    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.496849 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.496808    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.498135 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.498093    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-serving-cert\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.502523 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.502472    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwtqn\" (UniqueName: \"kubernetes.io/projected/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-kube-api-access-jwtqn\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:31:34.504249 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.504220    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8s96\" (UniqueName: \"kubernetes.io/projected/a2e3a1f9-cde2-4a79-ae33-9467155e2f14-kube-api-access-r8s96\") pod \"insights-operator-68d9dd94cf-hqvkg\" (UID: \"a2e3a1f9-cde2-4a79-ae33-9467155e2f14\") " pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.594103 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.594018    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-stats-auth\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:34.594103 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.594080    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-default-certificate\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:34.594326 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.594116    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:34.594326 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.594137    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:34.594326 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.594157    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lsbg\" (UniqueName: \"kubernetes.io/projected/8bbbac86-f54f-4df4-9839-ecbc28741a9f-kube-api-access-2lsbg\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:34.594326 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:34.594258    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 14 14:31:34.594577 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:34.594334    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs podName:8bbbac86-f54f-4df4-9839-ecbc28741a9f nodeName:}" failed. No retries permitted until 2026-07-14 14:31:35.094315823 +0000 UTC m=+72.760502237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs") pod "router-default-5965b849d4-jl74t" (UID: "8bbbac86-f54f-4df4-9839-ecbc28741a9f") : secret "router-metrics-certs-default" not found
Jul 14 14:31:34.594642 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:34.594606    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle podName:8bbbac86-f54f-4df4-9839-ecbc28741a9f nodeName:}" failed. No retries permitted until 2026-07-14 14:31:35.094541697 +0000 UTC m=+72.760728118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle") pod "router-default-5965b849d4-jl74t" (UID: "8bbbac86-f54f-4df4-9839-ecbc28741a9f") : configmap references non-existent config key: service-ca.crt
Jul 14 14:31:34.596908 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.596884    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-stats-auth\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:34.597100 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.597078    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-default-certificate\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:34.608744 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.608716    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lsbg\" (UniqueName: \"kubernetes.io/projected/8bbbac86-f54f-4df4-9839-ecbc28741a9f-kube-api-access-2lsbg\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:34.670440 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.670389    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg"
Jul 14 14:31:34.997494 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:34.997453    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:31:34.997666 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:34.997605    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 14 14:31:34.997708 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:34.997669    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls podName:a26ee4e3-b7fb-4ed4-9056-ae52a53b8667 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:35.997653183 +0000 UTC m=+73.663839604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-sgrwz" (UID: "a26ee4e3-b7fb-4ed4-9056-ae52a53b8667") : secret "cluster-monitoring-operator-tls" not found
Jul 14 14:31:35.098590 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:35.098547    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:35.098590 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:35.098589    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:35.098832 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:35.098717    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle podName:8bbbac86-f54f-4df4-9839-ecbc28741a9f nodeName:}" failed. No retries permitted until 2026-07-14 14:31:36.098703426 +0000 UTC m=+73.764889825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle") pod "router-default-5965b849d4-jl74t" (UID: "8bbbac86-f54f-4df4-9839-ecbc28741a9f") : configmap references non-existent config key: service-ca.crt
Jul 14 14:31:35.098832 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:35.098723    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 14 14:31:35.098832 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:35.098787    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs podName:8bbbac86-f54f-4df4-9839-ecbc28741a9f nodeName:}" failed. No retries permitted until 2026-07-14 14:31:36.098771008 +0000 UTC m=+73.764957407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs") pod "router-default-5965b849d4-jl74t" (UID: "8bbbac86-f54f-4df4-9839-ecbc28741a9f") : secret "router-metrics-certs-default" not found
Jul 14 14:31:35.598094 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:35.598062    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-hqvkg"]
Jul 14 14:31:35.602317 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:31:35.602292    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e3a1f9_cde2_4a79_ae33_9467155e2f14.slice/crio-c702b0ec753395d6d03fa5795ca44c871cda865b52a7c6e186363dc44dcca303 WatchSource:0}: Error finding container c702b0ec753395d6d03fa5795ca44c871cda865b52a7c6e186363dc44dcca303: Status 404 returned error can't find the container with id c702b0ec753395d6d03fa5795ca44c871cda865b52a7c6e186363dc44dcca303
Jul 14 14:31:36.003717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:36.003663    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:31:36.003874 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:36.003817    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 14 14:31:36.003934 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:36.003885    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls podName:a26ee4e3-b7fb-4ed4-9056-ae52a53b8667 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:38.003869393 +0000 UTC m=+75.670055804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-sgrwz" (UID: "a26ee4e3-b7fb-4ed4-9056-ae52a53b8667") : secret "cluster-monitoring-operator-tls" not found
Jul 14 14:31:36.105018 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:36.104963    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:36.105018 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:36.105017    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:36.105256 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:36.105126    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 14 14:31:36.105256 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:36.105151    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle podName:8bbbac86-f54f-4df4-9839-ecbc28741a9f nodeName:}" failed. No retries permitted until 2026-07-14 14:31:38.10513737 +0000 UTC m=+75.771323769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle") pod "router-default-5965b849d4-jl74t" (UID: "8bbbac86-f54f-4df4-9839-ecbc28741a9f") : configmap references non-existent config key: service-ca.crt
Jul 14 14:31:36.105256 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:36.105183    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs podName:8bbbac86-f54f-4df4-9839-ecbc28741a9f nodeName:}" failed. No retries permitted until 2026-07-14 14:31:38.105168432 +0000 UTC m=+75.771354831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs") pod "router-default-5965b849d4-jl74t" (UID: "8bbbac86-f54f-4df4-9839-ecbc28741a9f") : secret "router-metrics-certs-default" not found
Jul 14 14:31:36.173088 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:36.173034    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg" event={"ID":"a2e3a1f9-cde2-4a79-ae33-9467155e2f14","Type":"ContainerStarted","Data":"c702b0ec753395d6d03fa5795ca44c871cda865b52a7c6e186363dc44dcca303"}
Jul 14 14:31:36.174482 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:36.174455    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-84g4n" event={"ID":"667c8f62-f765-4395-9740-d0c0bc87102c","Type":"ContainerStarted","Data":"1c535b0c88503f721ccf000f8eb5dba1fa19859feb52d225fc7cad9181790eac"}
Jul 14 14:31:36.174607 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:36.174543    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:31:36.175915 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:36.175891    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rlbbv" event={"ID":"3febdf7d-4462-4194-a512-7c860fc44797","Type":"ContainerStarted","Data":"745da049adc7ac861ce88dfd4270b196b589bebc88085703d0d62535b30894ee"}
Jul 14 14:31:36.205293 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:36.205230    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rlbbv" podStartSLOduration=65.563501925 podStartE2EDuration="1m12.205209472s" totalImagesPullingTime="6.641707547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:31:36.204739003 +0000 UTC m=+73.870925423" watchObservedRunningTime="2026-07-14 14:31:36.205209472 +0000 UTC m=+73.871395894"
Jul 14 14:31:36.205478 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:36.205340    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-84g4n" podStartSLOduration=66.586970983 podStartE2EDuration="1m13.205333937s" totalImagesPullingTime="6.618362954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:31:36.189736708 +0000 UTC m=+73.855923129" watchObservedRunningTime="2026-07-14 14:31:36.205333937 +0000 UTC m=+73.871520357"
Jul 14 14:31:38.023578 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:38.023531    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:31:38.024020 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:38.023696    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 14 14:31:38.024020 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:38.023783    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls podName:a26ee4e3-b7fb-4ed4-9056-ae52a53b8667 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:42.023761052 +0000 UTC m=+79.689947455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-sgrwz" (UID: "a26ee4e3-b7fb-4ed4-9056-ae52a53b8667") : secret "cluster-monitoring-operator-tls" not found
Jul 14 14:31:38.123943 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:38.123910    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:38.124047 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:38.123968    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:38.124047 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:38.124025    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 14 14:31:38.124117 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:38.124101    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs podName:8bbbac86-f54f-4df4-9839-ecbc28741a9f nodeName:}" failed. No retries permitted until 2026-07-14 14:31:42.124082281 +0000 UTC m=+79.790268702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs") pod "router-default-5965b849d4-jl74t" (UID: "8bbbac86-f54f-4df4-9839-ecbc28741a9f") : secret "router-metrics-certs-default" not found
Jul 14 14:31:38.124117 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:38.124116    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle podName:8bbbac86-f54f-4df4-9839-ecbc28741a9f nodeName:}" failed. No retries permitted until 2026-07-14 14:31:42.124109771 +0000 UTC m=+79.790296170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle") pod "router-default-5965b849d4-jl74t" (UID: "8bbbac86-f54f-4df4-9839-ecbc28741a9f") : configmap references non-existent config key: service-ca.crt
Jul 14 14:31:38.182750 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:38.182718    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg" event={"ID":"a2e3a1f9-cde2-4a79-ae33-9467155e2f14","Type":"ContainerStarted","Data":"a4ba317e56966b57f83fccae0f3387ef85a2cf170113fc59dcca7898c12a42a7"}
Jul 14 14:31:38.200369 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:38.200290    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg" podStartSLOduration=1.691564262 podStartE2EDuration="4.200273s" totalImagesPullingTime="2.508708738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:31:38.199072745 +0000 UTC m=+75.865259167" watchObservedRunningTime="2026-07-14 14:31:38.200273 +0000 UTC m=+75.866459422"
Jul 14 14:31:39.526920 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.526891    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5sn8c"]
Jul 14 14:31:39.528941 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.528923    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:39.531530 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.531510    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 14 14:31:39.531639 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.531514    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 14 14:31:39.531837 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.531823    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-chg99\""
Jul 14 14:31:39.542600 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.542574    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5sn8c"]
Jul 14 14:31:39.634302 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.634271    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e0ef85f-65de-4d78-873a-9ce585c997cf-data-volume\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:39.634492 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.634338    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:39.634492 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.634374    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt868\" (UniqueName: \"kubernetes.io/projected/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-api-access-zt868\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:39.634492 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.634404    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e0ef85f-65de-4d78-873a-9ce585c997cf-crio-socket\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:39.634492 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.634459    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:39.735505 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.735470    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:39.735505 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.735503    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt868\" (UniqueName: \"kubernetes.io/projected/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-api-access-zt868\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:39.735698 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.735523    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e0ef85f-65de-4d78-873a-9ce585c997cf-crio-socket\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:39.735698 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.735558    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:39.735698 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.735595    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e0ef85f-65de-4d78-873a-9ce585c997cf-data-volume\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:39.735698 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:39.735605    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 14:31:39.735698 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.735641    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e0ef85f-65de-4d78-873a-9ce585c997cf-crio-socket\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:39.735698 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:39.735676    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm podName:2e0ef85f-65de-4d78-873a-9ce585c997cf nodeName:}" failed. No retries permitted until 2026-07-14 14:31:40.235657082 +0000 UTC m=+77.901843501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sn8c" (UID: "2e0ef85f-65de-4d78-873a-9ce585c997cf") : configmap "kube-rbac-proxy" not found
Jul 14 14:31:39.735882 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:39.735732    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 14:31:39.735882 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:39.735784    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls podName:2e0ef85f-65de-4d78-873a-9ce585c997cf nodeName:}" failed. No retries permitted until 2026-07-14 14:31:40.235768712 +0000 UTC m=+77.901955124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sn8c" (UID: "2e0ef85f-65de-4d78-873a-9ce585c997cf") : secret "insights-runtime-extractor-tls" not found
Jul 14 14:31:39.735947 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.735928    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e0ef85f-65de-4d78-873a-9ce585c997cf-data-volume\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:39.744260 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:39.744238    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt868\" (UniqueName: \"kubernetes.io/projected/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-api-access-zt868\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:40.241091 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:40.241047    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:40.241299 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:40.241110    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:40.241299 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:40.241183    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 14:31:40.241299 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:40.241217    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 14:31:40.241299 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:40.241264    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm podName:2e0ef85f-65de-4d78-873a-9ce585c997cf nodeName:}" failed. No retries permitted until 2026-07-14 14:31:41.241243607 +0000 UTC m=+78.907430005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sn8c" (UID: "2e0ef85f-65de-4d78-873a-9ce585c997cf") : configmap "kube-rbac-proxy" not found
Jul 14 14:31:40.241299 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:40.241282    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls podName:2e0ef85f-65de-4d78-873a-9ce585c997cf nodeName:}" failed. No retries permitted until 2026-07-14 14:31:41.241273842 +0000 UTC m=+78.907460246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sn8c" (UID: "2e0ef85f-65de-4d78-873a-9ce585c997cf") : secret "insights-runtime-extractor-tls" not found
Jul 14 14:31:40.903568 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:40.903539    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fl6rb_401faa4d-42da-4c1c-af1c-fe150089dfe0/dns-node-resolver/0.log"
Jul 14 14:31:41.249469 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:41.249403    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:41.249616 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:41.249511    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:41.249616 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:41.249534    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 14:31:41.249616 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:41.249607    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm podName:2e0ef85f-65de-4d78-873a-9ce585c997cf nodeName:}" failed. No retries permitted until 2026-07-14 14:31:43.249592058 +0000 UTC m=+80.915778474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sn8c" (UID: "2e0ef85f-65de-4d78-873a-9ce585c997cf") : configmap "kube-rbac-proxy" not found
Jul 14 14:31:41.249736 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:41.249655    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 14:31:41.249736 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:41.249711    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls podName:2e0ef85f-65de-4d78-873a-9ce585c997cf nodeName:}" failed. No retries permitted until 2026-07-14 14:31:43.249698313 +0000 UTC m=+80.915884712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sn8c" (UID: "2e0ef85f-65de-4d78-873a-9ce585c997cf") : secret "insights-runtime-extractor-tls" not found
Jul 14 14:31:42.056580 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:42.056543    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:31:42.057035 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:42.056728    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 14 14:31:42.057035 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:42.056814    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls podName:a26ee4e3-b7fb-4ed4-9056-ae52a53b8667 nodeName:}" failed. No retries permitted until 2026-07-14 14:31:50.056792425 +0000 UTC m=+87.722978844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-sgrwz" (UID: "a26ee4e3-b7fb-4ed4-9056-ae52a53b8667") : secret "cluster-monitoring-operator-tls" not found
Jul 14 14:31:42.103649 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:42.103617    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nqtwh_74295487-4d63-42cb-b367-fda447a5217b/node-ca/0.log"
Jul 14 14:31:42.157138 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:42.157099    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:42.157328 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:42.157149    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:42.157328 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:42.157267    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle podName:8bbbac86-f54f-4df4-9839-ecbc28741a9f nodeName:}" failed. No retries permitted until 2026-07-14 14:31:50.157251932 +0000 UTC m=+87.823438331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle") pod "router-default-5965b849d4-jl74t" (UID: "8bbbac86-f54f-4df4-9839-ecbc28741a9f") : configmap references non-existent config key: service-ca.crt
Jul 14 14:31:42.157328 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:42.157269    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 14 14:31:42.157509 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:42.157405    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs podName:8bbbac86-f54f-4df4-9839-ecbc28741a9f nodeName:}" failed. No retries permitted until 2026-07-14 14:31:50.157387392 +0000 UTC m=+87.823573798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs") pod "router-default-5965b849d4-jl74t" (UID: "8bbbac86-f54f-4df4-9839-ecbc28741a9f") : secret "router-metrics-certs-default" not found
Jul 14 14:31:42.703432 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:42.703383    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wxlrn_8b115d5b-3bdb-4b02-b747-a749b79536e4/serve-healthcheck-canary/0.log"
Jul 14 14:31:43.266719 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:43.266678    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:43.267088 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:43.266803    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:43.267088 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:43.266841    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 14:31:43.267088 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:43.266904    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls podName:2e0ef85f-65de-4d78-873a-9ce585c997cf nodeName:}" failed. No retries permitted until 2026-07-14 14:31:47.266889791 +0000 UTC m=+84.933076190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sn8c" (UID: "2e0ef85f-65de-4d78-873a-9ce585c997cf") : secret "insights-runtime-extractor-tls" not found
Jul 14 14:31:43.267088 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:43.266914    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 14:31:43.267088 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:43.266978    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm podName:2e0ef85f-65de-4d78-873a-9ce585c997cf nodeName:}" failed. No retries permitted until 2026-07-14 14:31:47.266959803 +0000 UTC m=+84.933146203 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sn8c" (UID: "2e0ef85f-65de-4d78-873a-9ce585c997cf") : configmap "kube-rbac-proxy" not found
Jul 14 14:31:47.298295 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:47.298248    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:47.298822 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:47.298311    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:47.298822 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:47.298377    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 14:31:47.298822 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:47.298409    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 14:31:47.298822 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:47.298472    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm podName:2e0ef85f-65de-4d78-873a-9ce585c997cf nodeName:}" failed. No retries permitted until 2026-07-14 14:31:55.298452961 +0000 UTC m=+92.964639372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sn8c" (UID: "2e0ef85f-65de-4d78-873a-9ce585c997cf") : configmap "kube-rbac-proxy" not found
Jul 14 14:31:47.298822 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:47.298488    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls podName:2e0ef85f-65de-4d78-873a-9ce585c997cf nodeName:}" failed. No retries permitted until 2026-07-14 14:31:55.298482018 +0000 UTC m=+92.964668416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sn8c" (UID: "2e0ef85f-65de-4d78-873a-9ce585c997cf") : secret "insights-runtime-extractor-tls" not found
Jul 14 14:31:50.121338 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:50.121303    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:31:50.121773 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:50.121473    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 14 14:31:50.121773 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:50.121538    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls podName:a26ee4e3-b7fb-4ed4-9056-ae52a53b8667 nodeName:}" failed. No retries permitted until 2026-07-14 14:32:06.121522888 +0000 UTC m=+103.787709303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-sgrwz" (UID: "a26ee4e3-b7fb-4ed4-9056-ae52a53b8667") : secret "cluster-monitoring-operator-tls" not found
Jul 14 14:31:50.221853 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:50.221825    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:50.222035 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:50.221873    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:31:50.222035 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:50.221984    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 14 14:31:50.222035 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:50.222011    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle podName:8bbbac86-f54f-4df4-9839-ecbc28741a9f nodeName:}" failed. No retries permitted until 2026-07-14 14:32:06.221992401 +0000 UTC m=+103.888178800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle") pod "router-default-5965b849d4-jl74t" (UID: "8bbbac86-f54f-4df4-9839-ecbc28741a9f") : configmap references non-existent config key: service-ca.crt
Jul 14 14:31:50.222205 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:50.222042    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs podName:8bbbac86-f54f-4df4-9839-ecbc28741a9f nodeName:}" failed. No retries permitted until 2026-07-14 14:32:06.222028131 +0000 UTC m=+103.888214531 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs") pod "router-default-5965b849d4-jl74t" (UID: "8bbbac86-f54f-4df4-9839-ecbc28741a9f") : secret "router-metrics-certs-default" not found
Jul 14 14:31:55.364705 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:55.364650    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:55.365098 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:55.364746    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:31:55.365098 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:55.364865    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 14:31:55.365098 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:31:55.364939    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls podName:2e0ef85f-65de-4d78-873a-9ce585c997cf nodeName:}" failed. No retries permitted until 2026-07-14 14:32:11.364919338 +0000 UTC m=+109.031105750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sn8c" (UID: "2e0ef85f-65de-4d78-873a-9ce585c997cf") : secret "insights-runtime-extractor-tls" not found
Jul 14 14:31:55.365302 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:31:55.365278    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e0ef85f-65de-4d78-873a-9ce585c997cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:32:03.529968 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:03.529925    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:32:03.532391 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:03.532370    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd676278-6891-49d4-881e-41f5b8029e62-metrics-tls\") pod \"dns-default-2f5v2\" (UID: \"dd676278-6891-49d4-881e-41f5b8029e62\") " pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:32:03.583353 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:03.583321    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x4st5\""
Jul 14 14:32:03.591566 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:03.591543    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:32:03.723300 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:03.723266    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2f5v2"]
Jul 14 14:32:03.726174 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:32:03.726144    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd676278_6891_49d4_881e_41f5b8029e62.slice/crio-5cd4e61d11db583cf6ef078479abfdc0c50c2bc2bba080c852972969f9a2ce1b WatchSource:0}: Error finding container 5cd4e61d11db583cf6ef078479abfdc0c50c2bc2bba080c852972969f9a2ce1b: Status 404 returned error can't find the container with id 5cd4e61d11db583cf6ef078479abfdc0c50c2bc2bba080c852972969f9a2ce1b
Jul 14 14:32:04.246724 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:04.246686    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2f5v2" event={"ID":"dd676278-6891-49d4-881e-41f5b8029e62","Type":"ContainerStarted","Data":"5cd4e61d11db583cf6ef078479abfdc0c50c2bc2bba080c852972969f9a2ce1b"}
Jul 14 14:32:06.153368 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:06.153279    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:32:06.155928 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:06.155904    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26ee4e3-b7fb-4ed4-9056-ae52a53b8667-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-sgrwz\" (UID: \"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:32:06.180269 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:06.180239    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"
Jul 14 14:32:06.253718 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:06.253676    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:32:06.253897 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:06.253731    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:32:06.254492 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:06.254462    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbbac86-f54f-4df4-9839-ecbc28741a9f-service-ca-bundle\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:32:06.254813 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:06.254786    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2f5v2" event={"ID":"dd676278-6891-49d4-881e-41f5b8029e62","Type":"ContainerStarted","Data":"f634c5d847129847c1bbf183d66f4edfe5103d3932b506e4c9eeffe0d5cdaa2d"}
Jul 14 14:32:06.254911 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:06.254844    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2f5v2" event={"ID":"dd676278-6891-49d4-881e-41f5b8029e62","Type":"ContainerStarted","Data":"04d8280861284ffaef7480f36a9daeb330faada70c12a35bdc1e2e63ac8c385c"}
Jul 14 14:32:06.255086 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:06.255067    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:32:06.256207 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:06.256188    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bbbac86-f54f-4df4-9839-ecbc28741a9f-metrics-certs\") pod \"router-default-5965b849d4-jl74t\" (UID: \"8bbbac86-f54f-4df4-9839-ecbc28741a9f\") " pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:32:06.271267 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:06.271226    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:32:06.272975 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:06.272894    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2f5v2" podStartSLOduration=65.201567748 podStartE2EDuration="1m7.272876738s" totalImagesPullingTime="2.07130899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:06.272580719 +0000 UTC m=+103.938767140" watchObservedRunningTime="2026-07-14 14:32:06.272876738 +0000 UTC m=+103.939063160"
Jul 14 14:32:06.312034 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:06.311996    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz"]
Jul 14 14:32:06.316364 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:32:06.316318    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26ee4e3_b7fb_4ed4_9056_ae52a53b8667.slice/crio-3d0aa3f693c51027fea55476dac4cd3bf1a85ca6fb10a2563fb2f74779f922d0 WatchSource:0}: Error finding container 3d0aa3f693c51027fea55476dac4cd3bf1a85ca6fb10a2563fb2f74779f922d0: Status 404 returned error can't find the container with id 3d0aa3f693c51027fea55476dac4cd3bf1a85ca6fb10a2563fb2f74779f922d0
Jul 14 14:32:06.406760 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:06.406671    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5965b849d4-jl74t"]
Jul 14 14:32:06.409488 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:32:06.409459    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbbac86_f54f_4df4_9839_ecbc28741a9f.slice/crio-370763679fe567f99f9a4a2e79d8f47cc0780ab29507dce8230e9a5d341969fe WatchSource:0}: Error finding container 370763679fe567f99f9a4a2e79d8f47cc0780ab29507dce8230e9a5d341969fe: Status 404 returned error can't find the container with id 370763679fe567f99f9a4a2e79d8f47cc0780ab29507dce8230e9a5d341969fe
Jul 14 14:32:07.181286 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:07.181258    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-84g4n"
Jul 14 14:32:07.260180 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:07.260143    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5965b849d4-jl74t" event={"ID":"8bbbac86-f54f-4df4-9839-ecbc28741a9f","Type":"ContainerStarted","Data":"8737ace1fbefeeedce259f239df65f2bff147468b2a76693dec116ef931b4625"}
Jul 14 14:32:07.260404 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:07.260209    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5965b849d4-jl74t" event={"ID":"8bbbac86-f54f-4df4-9839-ecbc28741a9f","Type":"ContainerStarted","Data":"370763679fe567f99f9a4a2e79d8f47cc0780ab29507dce8230e9a5d341969fe"}
Jul 14 14:32:07.261311 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:07.261285    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz" event={"ID":"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667","Type":"ContainerStarted","Data":"3d0aa3f693c51027fea55476dac4cd3bf1a85ca6fb10a2563fb2f74779f922d0"}
Jul 14 14:32:07.271702 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:07.271681    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:32:07.274330 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:07.274302    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:32:07.281218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:07.281166    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5965b849d4-jl74t" podStartSLOduration=33.28114587 podStartE2EDuration="33.28114587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:07.280709188 +0000 UTC m=+104.946895614" watchObservedRunningTime="2026-07-14 14:32:07.28114587 +0000 UTC m=+104.947332292"
Jul 14 14:32:08.265476 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:08.265436    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:32:08.266874 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:08.266839    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5965b849d4-jl74t"
Jul 14 14:32:09.270708 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:09.270670    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz" event={"ID":"a26ee4e3-b7fb-4ed4-9056-ae52a53b8667","Type":"ContainerStarted","Data":"62d10e4e7bf5a4ae0daccaed1868594a524bb36fb5ace405c630148a2854e6b9"}
Jul 14 14:32:09.291896 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:09.291837    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-sgrwz" podStartSLOduration=32.565739627 podStartE2EDuration="35.29181794s" totalImagesPullingTime="2.726078313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:09.289856995 +0000 UTC m=+106.956043417" watchObservedRunningTime="2026-07-14 14:32:09.29181794 +0000 UTC m=+106.958004360"
Jul 14 14:32:11.398366 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:11.398321    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:32:11.400833 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:11.400787    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e0ef85f-65de-4d78-873a-9ce585c997cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sn8c\" (UID: \"2e0ef85f-65de-4d78-873a-9ce585c997cf\") " pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:32:11.637300 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:11.637263    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5sn8c"
Jul 14 14:32:11.770625 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:11.770589    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5sn8c"]
Jul 14 14:32:11.775368 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:32:11.775343    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0ef85f_65de_4d78_873a_9ce585c997cf.slice/crio-2ae366c1b55154052f92dc8b72c99824a0ec21764026627bdcbd49a3dc90e1ff WatchSource:0}: Error finding container 2ae366c1b55154052f92dc8b72c99824a0ec21764026627bdcbd49a3dc90e1ff: Status 404 returned error can't find the container with id 2ae366c1b55154052f92dc8b72c99824a0ec21764026627bdcbd49a3dc90e1ff
Jul 14 14:32:12.283356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:12.283308    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5sn8c" event={"ID":"2e0ef85f-65de-4d78-873a-9ce585c997cf","Type":"ContainerStarted","Data":"2ae366c1b55154052f92dc8b72c99824a0ec21764026627bdcbd49a3dc90e1ff"}
Jul 14 14:32:13.454959 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.454920    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"]
Jul 14 14:32:13.457312 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.457292    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"
Jul 14 14:32:13.462891 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.462864    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 14 14:32:13.463488 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.463173    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 14 14:32:13.463488 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.463286    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 14 14:32:13.463488 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.463362    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 14 14:32:13.469893 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.469849    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"]
Jul 14 14:32:13.564554 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.564511    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-tprcb"]
Jul 14 14:32:13.566548 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.566528    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-tprcb"
Jul 14 14:32:13.572879 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.572855    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vgjlx\""
Jul 14 14:32:13.575439 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.575393    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 14 14:32:13.575544 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.575520    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 14 14:32:13.581297 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.581269    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cb956c59d-c49sp"]
Jul 14 14:32:13.583018 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.582999    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.596799 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.596766    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 14 14:32:13.596799 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.596791    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 14 14:32:13.596963 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.596823    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 14 14:32:13.596963 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.596768    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m9z97\""
Jul 14 14:32:13.607226 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.607194    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-tprcb"]
Jul 14 14:32:13.616928 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.616907    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1d39a6b5-8936-43df-9b01-dfdd7da78dc8-klusterlet-config\") pod \"klusterlet-addon-workmgr-75dfc6c4d7-mc58d\" (UID: \"1d39a6b5-8936-43df-9b01-dfdd7da78dc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"
Jul 14 14:32:13.617038 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.616960    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d39a6b5-8936-43df-9b01-dfdd7da78dc8-tmp\") pod \"klusterlet-addon-workmgr-75dfc6c4d7-mc58d\" (UID: \"1d39a6b5-8936-43df-9b01-dfdd7da78dc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"
Jul 14 14:32:13.617089 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.617065    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcx8n\" (UniqueName: \"kubernetes.io/projected/1d39a6b5-8936-43df-9b01-dfdd7da78dc8-kube-api-access-dcx8n\") pod \"klusterlet-addon-workmgr-75dfc6c4d7-mc58d\" (UID: \"1d39a6b5-8936-43df-9b01-dfdd7da78dc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"
Jul 14 14:32:13.634861 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.634458    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 14 14:32:13.641039 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.641013    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cb956c59d-c49sp"]
Jul 14 14:32:13.642156 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.642134    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bqmmx"]
Jul 14 14:32:13.644347 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.644323    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bqmmx"
Jul 14 14:32:13.647370 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.647352    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-m4699\""
Jul 14 14:32:13.647503 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.647355    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 14 14:32:13.668930 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.668902    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bqmmx"]
Jul 14 14:32:13.712770 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.712695    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cb956c59d-c49sp"]
Jul 14 14:32:13.712950 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:32:13.712929    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-bdpgt registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6cb956c59d-c49sp" podUID="ef729a8c-5236-49e5-8b2d-86d06345e4ce"
Jul 14 14:32:13.718087 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.718059    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcx8n\" (UniqueName: \"kubernetes.io/projected/1d39a6b5-8936-43df-9b01-dfdd7da78dc8-kube-api-access-dcx8n\") pod \"klusterlet-addon-workmgr-75dfc6c4d7-mc58d\" (UID: \"1d39a6b5-8936-43df-9b01-dfdd7da78dc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"
Jul 14 14:32:13.718213 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.718105    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef729a8c-5236-49e5-8b2d-86d06345e4ce-installation-pull-secrets\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.718213 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.718136    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/745b2f3b-5fd7-4ca7-885d-c5a465c91a6f-nginx-conf\") pod \"networking-console-plugin-6cf856758c-tprcb\" (UID: \"745b2f3b-5fd7-4ca7-885d-c5a465c91a6f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tprcb"
Jul 14 14:32:13.718213 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.718161    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-bound-sa-token\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.718213 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.718191    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef729a8c-5236-49e5-8b2d-86d06345e4ce-trusted-ca\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.718386 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.718231    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1d39a6b5-8936-43df-9b01-dfdd7da78dc8-klusterlet-config\") pod \"klusterlet-addon-workmgr-75dfc6c4d7-mc58d\" (UID: \"1d39a6b5-8936-43df-9b01-dfdd7da78dc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"
Jul 14 14:32:13.718386 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.718258    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdpgt\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-kube-api-access-bdpgt\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.718386 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.718302    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef729a8c-5236-49e5-8b2d-86d06345e4ce-ca-trust-extracted\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.718386 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.718330    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/745b2f3b-5fd7-4ca7-885d-c5a465c91a6f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tprcb\" (UID: \"745b2f3b-5fd7-4ca7-885d-c5a465c91a6f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tprcb"
Jul 14 14:32:13.718386 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.718357    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d39a6b5-8936-43df-9b01-dfdd7da78dc8-tmp\") pod \"klusterlet-addon-workmgr-75dfc6c4d7-mc58d\" (UID: \"1d39a6b5-8936-43df-9b01-dfdd7da78dc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"
Jul 14 14:32:13.718595 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.718388    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-registry-tls\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.718595 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.718452    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef729a8c-5236-49e5-8b2d-86d06345e4ce-registry-certificates\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.718595 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.718485    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef729a8c-5236-49e5-8b2d-86d06345e4ce-image-registry-private-configuration\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.718937 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.718909    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d39a6b5-8936-43df-9b01-dfdd7da78dc8-tmp\") pod \"klusterlet-addon-workmgr-75dfc6c4d7-mc58d\" (UID: \"1d39a6b5-8936-43df-9b01-dfdd7da78dc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"
Jul 14 14:32:13.721392 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.721371    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1d39a6b5-8936-43df-9b01-dfdd7da78dc8-klusterlet-config\") pod \"klusterlet-addon-workmgr-75dfc6c4d7-mc58d\" (UID: \"1d39a6b5-8936-43df-9b01-dfdd7da78dc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"
Jul 14 14:32:13.730186 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.730159    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcx8n\" (UniqueName: \"kubernetes.io/projected/1d39a6b5-8936-43df-9b01-dfdd7da78dc8-kube-api-access-dcx8n\") pod \"klusterlet-addon-workmgr-75dfc6c4d7-mc58d\" (UID: \"1d39a6b5-8936-43df-9b01-dfdd7da78dc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"
Jul 14 14:32:13.774075 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.774043    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"
Jul 14 14:32:13.819814 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.819781    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef729a8c-5236-49e5-8b2d-86d06345e4ce-trusted-ca\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.819938 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.819832    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdpgt\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-kube-api-access-bdpgt\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.819938 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.819864    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef729a8c-5236-49e5-8b2d-86d06345e4ce-ca-trust-extracted\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.819938 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.819885    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/745b2f3b-5fd7-4ca7-885d-c5a465c91a6f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tprcb\" (UID: \"745b2f3b-5fd7-4ca7-885d-c5a465c91a6f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tprcb"
Jul 14 14:32:13.819938 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.819906    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/29a1fc9f-e7cf-46f6-879d-957d1a46253c-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-bqmmx\" (UID: \"29a1fc9f-e7cf-46f6-879d-957d1a46253c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bqmmx"
Jul 14 14:32:13.819938 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.819924    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-registry-tls\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.820459 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.819945    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef729a8c-5236-49e5-8b2d-86d06345e4ce-registry-certificates\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.820459 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.819964    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef729a8c-5236-49e5-8b2d-86d06345e4ce-image-registry-private-configuration\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.820459 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.819999    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef729a8c-5236-49e5-8b2d-86d06345e4ce-installation-pull-secrets\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.820459 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.820015    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/745b2f3b-5fd7-4ca7-885d-c5a465c91a6f-nginx-conf\") pod \"networking-console-plugin-6cf856758c-tprcb\" (UID: \"745b2f3b-5fd7-4ca7-885d-c5a465c91a6f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tprcb"
Jul 14 14:32:13.820459 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.820028    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-bound-sa-token\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.821261 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.821101    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef729a8c-5236-49e5-8b2d-86d06345e4ce-trusted-ca\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.821261 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.821211    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/745b2f3b-5fd7-4ca7-885d-c5a465c91a6f-nginx-conf\") pod \"networking-console-plugin-6cf856758c-tprcb\" (UID: \"745b2f3b-5fd7-4ca7-885d-c5a465c91a6f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tprcb"
Jul 14 14:32:13.821872 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.821847    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef729a8c-5236-49e5-8b2d-86d06345e4ce-registry-certificates\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.824405 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.824050    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef729a8c-5236-49e5-8b2d-86d06345e4ce-image-registry-private-configuration\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.825160 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.824576    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-registry-tls\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.825160 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.825103    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/745b2f3b-5fd7-4ca7-885d-c5a465c91a6f-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tprcb\" (UID: \"745b2f3b-5fd7-4ca7-885d-c5a465c91a6f\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tprcb"
Jul 14 14:32:13.825489 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.825468    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef729a8c-5236-49e5-8b2d-86d06345e4ce-ca-trust-extracted\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.826141 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.826110    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef729a8c-5236-49e5-8b2d-86d06345e4ce-installation-pull-secrets\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.835906 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.835870    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-bound-sa-token\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.840040 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.839952    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdpgt\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-kube-api-access-bdpgt\") pod \"image-registry-6cb956c59d-c49sp\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") " pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:13.876997 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.876971    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-tprcb"
Jul 14 14:32:13.921150 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.921004    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/29a1fc9f-e7cf-46f6-879d-957d1a46253c-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-bqmmx\" (UID: \"29a1fc9f-e7cf-46f6-879d-957d1a46253c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bqmmx"
Jul 14 14:32:13.924118 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.924054    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/29a1fc9f-e7cf-46f6-879d-957d1a46253c-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-bqmmx\" (UID: \"29a1fc9f-e7cf-46f6-879d-957d1a46253c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bqmmx"
Jul 14 14:32:13.924896 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.924871    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"]
Jul 14 14:32:13.927788 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:32:13.927758    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d39a6b5_8936_43df_9b01_dfdd7da78dc8.slice/crio-b90997f208d6bf74fa2503c1cfd7344769fb76fdde9844e73e612f802080ac0f WatchSource:0}: Error finding container b90997f208d6bf74fa2503c1cfd7344769fb76fdde9844e73e612f802080ac0f: Status 404 returned error can't find the container with id b90997f208d6bf74fa2503c1cfd7344769fb76fdde9844e73e612f802080ac0f
Jul 14 14:32:13.954575 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:13.954547    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bqmmx"
Jul 14 14:32:14.028799 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.028768    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-tprcb"]
Jul 14 14:32:14.032264 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:32:14.032234    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745b2f3b_5fd7_4ca7_885d_c5a465c91a6f.slice/crio-32a57507ca3048183464db6689f134a6986b7aa2e8737f497afd8390fbbf1616 WatchSource:0}: Error finding container 32a57507ca3048183464db6689f134a6986b7aa2e8737f497afd8390fbbf1616: Status 404 returned error can't find the container with id 32a57507ca3048183464db6689f134a6986b7aa2e8737f497afd8390fbbf1616
Jul 14 14:32:14.103068 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.103030    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bqmmx"]
Jul 14 14:32:14.291255 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.291161    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-tprcb" event={"ID":"745b2f3b-5fd7-4ca7-885d-c5a465c91a6f","Type":"ContainerStarted","Data":"32a57507ca3048183464db6689f134a6986b7aa2e8737f497afd8390fbbf1616"}
Jul 14 14:32:14.292626 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.292593    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:14.292626 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.292604    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d" event={"ID":"1d39a6b5-8936-43df-9b01-dfdd7da78dc8","Type":"ContainerStarted","Data":"b90997f208d6bf74fa2503c1cfd7344769fb76fdde9844e73e612f802080ac0f"}
Jul 14 14:32:14.302777 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.302755    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:14.425831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.425794    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdpgt\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-kube-api-access-bdpgt\") pod \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") "
Jul 14 14:32:14.426021 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.425882    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef729a8c-5236-49e5-8b2d-86d06345e4ce-ca-trust-extracted\") pod \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") "
Jul 14 14:32:14.426021 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.425918    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef729a8c-5236-49e5-8b2d-86d06345e4ce-trusted-ca\") pod \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") "
Jul 14 14:32:14.426021 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.425939    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef729a8c-5236-49e5-8b2d-86d06345e4ce-registry-certificates\") pod \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") "
Jul 14 14:32:14.426021 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.425967    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-bound-sa-token\") pod \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") "
Jul 14 14:32:14.426021 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.426003    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-registry-tls\") pod \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") "
Jul 14 14:32:14.426268 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.426034    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef729a8c-5236-49e5-8b2d-86d06345e4ce-installation-pull-secrets\") pod \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") "
Jul 14 14:32:14.426268 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.426064    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef729a8c-5236-49e5-8b2d-86d06345e4ce-image-registry-private-configuration\") pod \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\" (UID: \"ef729a8c-5236-49e5-8b2d-86d06345e4ce\") "
Jul 14 14:32:14.426268 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.426166    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef729a8c-5236-49e5-8b2d-86d06345e4ce-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ef729a8c-5236-49e5-8b2d-86d06345e4ce" (UID: "ef729a8c-5236-49e5-8b2d-86d06345e4ce"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:32:14.426446 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.426382    2586 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef729a8c-5236-49e5-8b2d-86d06345e4ce-ca-trust-extracted\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:32:14.426446 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.426399    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef729a8c-5236-49e5-8b2d-86d06345e4ce-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ef729a8c-5236-49e5-8b2d-86d06345e4ce" (UID: "ef729a8c-5236-49e5-8b2d-86d06345e4ce"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:32:14.426864 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.426828    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef729a8c-5236-49e5-8b2d-86d06345e4ce-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ef729a8c-5236-49e5-8b2d-86d06345e4ce" (UID: "ef729a8c-5236-49e5-8b2d-86d06345e4ce"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:32:14.428747 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.428720    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-kube-api-access-bdpgt" (OuterVolumeSpecName: "kube-api-access-bdpgt") pod "ef729a8c-5236-49e5-8b2d-86d06345e4ce" (UID: "ef729a8c-5236-49e5-8b2d-86d06345e4ce"). InnerVolumeSpecName "kube-api-access-bdpgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:32:14.429252 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.429152    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ef729a8c-5236-49e5-8b2d-86d06345e4ce" (UID: "ef729a8c-5236-49e5-8b2d-86d06345e4ce"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:32:14.429252 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.429143    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef729a8c-5236-49e5-8b2d-86d06345e4ce-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ef729a8c-5236-49e5-8b2d-86d06345e4ce" (UID: "ef729a8c-5236-49e5-8b2d-86d06345e4ce"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:32:14.429252 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.429207    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef729a8c-5236-49e5-8b2d-86d06345e4ce-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ef729a8c-5236-49e5-8b2d-86d06345e4ce" (UID: "ef729a8c-5236-49e5-8b2d-86d06345e4ce"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:32:14.429468 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.429449    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ef729a8c-5236-49e5-8b2d-86d06345e4ce" (UID: "ef729a8c-5236-49e5-8b2d-86d06345e4ce"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:32:14.526977 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.526942    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdpgt\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-kube-api-access-bdpgt\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:32:14.526977 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.526975    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef729a8c-5236-49e5-8b2d-86d06345e4ce-trusted-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:32:14.527436 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.526988    2586 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef729a8c-5236-49e5-8b2d-86d06345e4ce-registry-certificates\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:32:14.527436 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.527000    2586 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-bound-sa-token\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:32:14.527436 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.527011    2586 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef729a8c-5236-49e5-8b2d-86d06345e4ce-registry-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:32:14.527436 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.527022    2586 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef729a8c-5236-49e5-8b2d-86d06345e4ce-installation-pull-secrets\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:32:14.527436 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:14.527051    2586 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef729a8c-5236-49e5-8b2d-86d06345e4ce-image-registry-private-configuration\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:32:15.295320 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:15.295286    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb956c59d-c49sp"
Jul 14 14:32:15.327049 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:15.327010    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cb956c59d-c49sp"]
Jul 14 14:32:15.330608 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:15.330578    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6cb956c59d-c49sp"]
Jul 14 14:32:15.339914 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:32:15.339882    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a1fc9f_e7cf_46f6_879d_957d1a46253c.slice/crio-b7887f6545ceb378b804e29315b969531f87fdda88776ef688ac6ce277c99f2a WatchSource:0}: Error finding container b7887f6545ceb378b804e29315b969531f87fdda88776ef688ac6ce277c99f2a: Status 404 returned error can't find the container with id b7887f6545ceb378b804e29315b969531f87fdda88776ef688ac6ce277c99f2a
Jul 14 14:32:16.263911 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:16.263816    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2f5v2"
Jul 14 14:32:16.300632 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:16.300594    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bqmmx" event={"ID":"29a1fc9f-e7cf-46f6-879d-957d1a46253c","Type":"ContainerStarted","Data":"b7887f6545ceb378b804e29315b969531f87fdda88776ef688ac6ce277c99f2a"}
Jul 14 14:32:16.876392 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:16.876357    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef729a8c-5236-49e5-8b2d-86d06345e4ce" path="/var/lib/kubelet/pods/ef729a8c-5236-49e5-8b2d-86d06345e4ce/volumes"
Jul 14 14:32:17.305871 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:17.305833    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5sn8c" event={"ID":"2e0ef85f-65de-4d78-873a-9ce585c997cf","Type":"ContainerStarted","Data":"55f7d2f8fdff0645e8e39a6dca575f56f60f47a8e0646e29c6e5509550e3cdcf"}
Jul 14 14:32:20.320900 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.320862    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bqmmx" event={"ID":"29a1fc9f-e7cf-46f6-879d-957d1a46253c","Type":"ContainerStarted","Data":"8b74dc799d343871b0e9b89157b906a8af9c2ae9a4996c6132b1f3f8dc52e442"}
Jul 14 14:32:20.321536 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.320952    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bqmmx"
Jul 14 14:32:20.323193 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.323147    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5sn8c" event={"ID":"2e0ef85f-65de-4d78-873a-9ce585c997cf","Type":"ContainerStarted","Data":"30018b7b0ff1fb18204d4f389ca27362a7e05c6c13a3e59b8aa4db7f5f6175e9"}
Jul 14 14:32:20.324761 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.324727    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-tprcb" event={"ID":"745b2f3b-5fd7-4ca7-885d-c5a465c91a6f","Type":"ContainerStarted","Data":"a73d9d11b484b787416b95af9d5058931240adc3374a52d3a98eea656a311131"}
Jul 14 14:32:20.326497 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.326469    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bqmmx"
Jul 14 14:32:20.326815 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.326795    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d" event={"ID":"1d39a6b5-8936-43df-9b01-dfdd7da78dc8","Type":"ContainerStarted","Data":"872bbc3770d5ad556c26efdce0545eb6491163915f29bc125c73d6edffc7b6f7"}
Jul 14 14:32:20.327068 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.327037    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"
Jul 14 14:32:20.328488 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.328470    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d"
Jul 14 14:32:20.337317 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.337266    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bqmmx" podStartSLOduration=3.259454426 podStartE2EDuration="7.337251271s" totalImagesPullingTime="4.077796845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:20.336468658 +0000 UTC m=+118.002655080" watchObservedRunningTime="2026-07-14 14:32:20.337251271 +0000 UTC m=+118.003437710"
Jul 14 14:32:20.352906 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.352847    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-tprcb" podStartSLOduration=1.965505318 podStartE2EDuration="7.352832233s" totalImagesPullingTime="5.387326915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:20.351660326 +0000 UTC m=+118.017846747" watchObservedRunningTime="2026-07-14 14:32:20.352832233 +0000 UTC m=+118.019018633"
Jul 14 14:32:20.391045 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.390990    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75dfc6c4d7-mc58d" podStartSLOduration=1.7797837749999998 podStartE2EDuration="7.390968007s" totalImagesPullingTime="5.611184232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:20.390394746 +0000 UTC m=+118.056581168" watchObservedRunningTime="2026-07-14 14:32:20.390968007 +0000 UTC m=+118.057154430"
Jul 14 14:32:20.637478 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.637355    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-z6lcx"]
Jul 14 14:32:20.640609 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.640586    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx"
Jul 14 14:32:20.654021 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.651824    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 14 14:32:20.654021 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.651972    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 14 14:32:20.654021 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.652100    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8pmxt\""
Jul 14 14:32:20.654021 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.652438    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 14 14:32:20.654021 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.653937    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-z6lcx"]
Jul 14 14:32:20.678634 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.678598    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxnrc\" (UniqueName: \"kubernetes.io/projected/16cee93e-ca72-44e1-a79b-670955bea80b-kube-api-access-gxnrc\") pod \"prometheus-operator-64785b6675-z6lcx\" (UID: \"16cee93e-ca72-44e1-a79b-670955bea80b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx"
Jul 14 14:32:20.678793 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.678689    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16cee93e-ca72-44e1-a79b-670955bea80b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-z6lcx\" (UID: \"16cee93e-ca72-44e1-a79b-670955bea80b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx"
Jul 14 14:32:20.678793 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.678719    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16cee93e-ca72-44e1-a79b-670955bea80b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-z6lcx\" (UID: \"16cee93e-ca72-44e1-a79b-670955bea80b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx"
Jul 14 14:32:20.678793 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.678748    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16cee93e-ca72-44e1-a79b-670955bea80b-metrics-client-ca\") pod \"prometheus-operator-64785b6675-z6lcx\" (UID: \"16cee93e-ca72-44e1-a79b-670955bea80b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx"
Jul 14 14:32:20.779519 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.779489    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16cee93e-ca72-44e1-a79b-670955bea80b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-z6lcx\" (UID: \"16cee93e-ca72-44e1-a79b-670955bea80b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx"
Jul 14 14:32:20.779700 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.779523    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16cee93e-ca72-44e1-a79b-670955bea80b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-z6lcx\" (UID: \"16cee93e-ca72-44e1-a79b-670955bea80b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx"
Jul 14 14:32:20.779700 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.779547    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16cee93e-ca72-44e1-a79b-670955bea80b-metrics-client-ca\") pod \"prometheus-operator-64785b6675-z6lcx\" (UID: \"16cee93e-ca72-44e1-a79b-670955bea80b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx"
Jul 14 14:32:20.779800 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.779694    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxnrc\" (UniqueName: \"kubernetes.io/projected/16cee93e-ca72-44e1-a79b-670955bea80b-kube-api-access-gxnrc\") pod \"prometheus-operator-64785b6675-z6lcx\" (UID: \"16cee93e-ca72-44e1-a79b-670955bea80b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx"
Jul 14 14:32:20.780251 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.780224    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16cee93e-ca72-44e1-a79b-670955bea80b-metrics-client-ca\") pod \"prometheus-operator-64785b6675-z6lcx\" (UID: \"16cee93e-ca72-44e1-a79b-670955bea80b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx"
Jul 14 14:32:20.782027 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.782001    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16cee93e-ca72-44e1-a79b-670955bea80b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-z6lcx\" (UID: \"16cee93e-ca72-44e1-a79b-670955bea80b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx"
Jul 14 14:32:20.782198 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.782178    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16cee93e-ca72-44e1-a79b-670955bea80b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-z6lcx\" (UID: \"16cee93e-ca72-44e1-a79b-670955bea80b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx"
Jul 14 14:32:20.789176 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.789153    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxnrc\" (UniqueName: \"kubernetes.io/projected/16cee93e-ca72-44e1-a79b-670955bea80b-kube-api-access-gxnrc\") pod \"prometheus-operator-64785b6675-z6lcx\" (UID: \"16cee93e-ca72-44e1-a79b-670955bea80b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx"
Jul 14 14:32:20.961694 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:20.961663    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx"
Jul 14 14:32:21.091157 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:21.091126    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-z6lcx"]
Jul 14 14:32:21.094450 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:32:21.094405    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16cee93e_ca72_44e1_a79b_670955bea80b.slice/crio-92dc274417d0d12baf666f445a9e3086e74dba90585d5734834f7786ca59c025 WatchSource:0}: Error finding container 92dc274417d0d12baf666f445a9e3086e74dba90585d5734834f7786ca59c025: Status 404 returned error can't find the container with id 92dc274417d0d12baf666f445a9e3086e74dba90585d5734834f7786ca59c025
Jul 14 14:32:21.331080 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:21.330984    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx" event={"ID":"16cee93e-ca72-44e1-a79b-670955bea80b","Type":"ContainerStarted","Data":"92dc274417d0d12baf666f445a9e3086e74dba90585d5734834f7786ca59c025"}
Jul 14 14:32:24.345581 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:24.345539    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5sn8c" event={"ID":"2e0ef85f-65de-4d78-873a-9ce585c997cf","Type":"ContainerStarted","Data":"33bcd5e2dc8c82f067e50b593979ee684cd9f523a4ace856501a8e93a4fe6b29"}
Jul 14 14:32:24.347269 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:24.347241    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx" event={"ID":"16cee93e-ca72-44e1-a79b-670955bea80b","Type":"ContainerStarted","Data":"e2a807d3f48232f435cbad16b41a403a3495a1dcc1560dc17b346ad47cbecb56"}
Jul 14 14:32:24.347471 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:24.347288    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx" event={"ID":"16cee93e-ca72-44e1-a79b-670955bea80b","Type":"ContainerStarted","Data":"c5d4eacaabb1448324404e898e005ddc7a3268c6d8021041ee77c0df88ef11fa"}
Jul 14 14:32:24.365194 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:24.365139    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5sn8c" podStartSLOduration=33.443647812 podStartE2EDuration="45.365126126s" totalImagesPullingTime="11.921478314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:31:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:24.363774865 +0000 UTC m=+122.029961286" watchObservedRunningTime="2026-07-14 14:32:24.365126126 +0000 UTC m=+122.031312546"
Jul 14 14:32:24.381547 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:24.381492    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-z6lcx" podStartSLOduration=1.636208465 podStartE2EDuration="4.3814761s" totalImagesPullingTime="2.745267635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:24.380651655 +0000 UTC m=+122.046838076" watchObservedRunningTime="2026-07-14 14:32:24.3814761 +0000 UTC m=+122.047662520"
Jul 14 14:32:25.977568 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:25.977533    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"]
Jul 14 14:32:25.980008 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:25.979980    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:25.980580 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:25.980556    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-56qqr"]
Jul 14 14:32:25.982568 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:25.982548    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:25.983008 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:25.982989    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 14 14:32:25.983212 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:25.983196    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 14 14:32:25.983445 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:25.983427    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bmrzx\""
Jul 14 14:32:25.985336 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:25.985314    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8j6fw\""
Jul 14 14:32:25.985467 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:25.985338    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 14 14:32:25.985467 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:25.985360    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 14 14:32:25.985467 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:25.985452    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 14 14:32:25.991505 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:25.991474    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"]
Jul 14 14:32:25.995879 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:25.995853    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-56qqr"]
Jul 14 14:32:26.018601 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.018569    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mfkkg"]
Jul 14 14:32:26.021074 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.021057    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.024201 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.024172    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 14 14:32:26.024320 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.024244    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 14 14:32:26.024584 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.024556    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 14 14:32:26.024680 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.024601    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gfn6m\""
Jul 14 14:32:26.025405 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.025383    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41817d03-f7be-4b25-991e-7deaaf44b9dc-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-vjcnd\" (UID: \"41817d03-f7be-4b25-991e-7deaaf44b9dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:26.025521 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.025504    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.025566 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.025535    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.025600 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.025565    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.025600 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.025586    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41817d03-f7be-4b25-991e-7deaaf44b9dc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-vjcnd\" (UID: \"41817d03-f7be-4b25-991e-7deaaf44b9dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:26.025700 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.025626    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.025731 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.025707    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8njnw\" (UniqueName: \"kubernetes.io/projected/41817d03-f7be-4b25-991e-7deaaf44b9dc-kube-api-access-8njnw\") pod \"openshift-state-metrics-84bd68b896-vjcnd\" (UID: \"41817d03-f7be-4b25-991e-7deaaf44b9dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:26.025763 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.025736    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf2gh\" (UniqueName: \"kubernetes.io/projected/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-api-access-qf2gh\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.025797 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.025763    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41817d03-f7be-4b25-991e-7deaaf44b9dc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-vjcnd\" (UID: \"41817d03-f7be-4b25-991e-7deaaf44b9dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:26.025830 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.025794    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.127045 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127006    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.127045 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127049    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.127318 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127076    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.127318 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127095    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.127318 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127119    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41817d03-f7be-4b25-991e-7deaaf44b9dc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-vjcnd\" (UID: \"41817d03-f7be-4b25-991e-7deaaf44b9dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:26.127318 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127147    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.127318 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:32:26.127201    2586 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 14 14:32:26.127318 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127228    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d50dbe94-4da7-4160-ab95-7f028a88d5b6-sys\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.127318 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:32:26.127243    2586 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 14 14:32:26.127318 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127267    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-wtmp\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.127318 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:32:26.127286    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-state-metrics-tls podName:6e84e3d2-b8b1-48e4-81a5-6d5d28d42898 nodeName:}" failed. No retries permitted until 2026-07-14 14:32:26.627262766 +0000 UTC m=+124.293449178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-56qqr" (UID: "6e84e3d2-b8b1-48e4-81a5-6d5d28d42898") : secret "kube-state-metrics-tls" not found
Jul 14 14:32:26.127318 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127325    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8njnw\" (UniqueName: \"kubernetes.io/projected/41817d03-f7be-4b25-991e-7deaaf44b9dc-kube-api-access-8njnw\") pod \"openshift-state-metrics-84bd68b896-vjcnd\" (UID: \"41817d03-f7be-4b25-991e-7deaaf44b9dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:26.127899 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:32:26.127335    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41817d03-f7be-4b25-991e-7deaaf44b9dc-openshift-state-metrics-tls podName:41817d03-f7be-4b25-991e-7deaaf44b9dc nodeName:}" failed. No retries permitted until 2026-07-14 14:32:26.627324851 +0000 UTC m=+124.293511251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/41817d03-f7be-4b25-991e-7deaaf44b9dc-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-vjcnd" (UID: "41817d03-f7be-4b25-991e-7deaaf44b9dc") : secret "openshift-state-metrics-tls" not found
Jul 14 14:32:26.127899 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127384    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d50dbe94-4da7-4160-ab95-7f028a88d5b6-root\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.127899 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127444    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-tls\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.127899 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127477    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf2gh\" (UniqueName: \"kubernetes.io/projected/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-api-access-qf2gh\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.127899 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127520    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41817d03-f7be-4b25-991e-7deaaf44b9dc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-vjcnd\" (UID: \"41817d03-f7be-4b25-991e-7deaaf44b9dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:26.127899 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127537    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.127899 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127555    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.127899 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127586    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d50dbe94-4da7-4160-ab95-7f028a88d5b6-metrics-client-ca\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.127899 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127621    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41817d03-f7be-4b25-991e-7deaaf44b9dc-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-vjcnd\" (UID: \"41817d03-f7be-4b25-991e-7deaaf44b9dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:26.127899 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127645    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d29b\" (UniqueName: \"kubernetes.io/projected/d50dbe94-4da7-4160-ab95-7f028a88d5b6-kube-api-access-8d29b\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.127899 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127691    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-textfile\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.127899 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127723    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.127899 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127844    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.127899 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.127890    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.128312 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.128198    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41817d03-f7be-4b25-991e-7deaaf44b9dc-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-vjcnd\" (UID: \"41817d03-f7be-4b25-991e-7deaaf44b9dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:26.129945 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.129921    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41817d03-f7be-4b25-991e-7deaaf44b9dc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-vjcnd\" (UID: \"41817d03-f7be-4b25-991e-7deaaf44b9dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:26.130121 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.130105    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.136319 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.136286    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8njnw\" (UniqueName: \"kubernetes.io/projected/41817d03-f7be-4b25-991e-7deaaf44b9dc-kube-api-access-8njnw\") pod \"openshift-state-metrics-84bd68b896-vjcnd\" (UID: \"41817d03-f7be-4b25-991e-7deaaf44b9dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:26.136588 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.136566    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf2gh\" (UniqueName: \"kubernetes.io/projected/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-api-access-qf2gh\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.228310 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.228222    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-textfile\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.228310 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.228263    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.228549 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.228458    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.228549 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.228498    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d50dbe94-4da7-4160-ab95-7f028a88d5b6-sys\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.228549 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.228527    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-wtmp\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.228706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.228580    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-textfile\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.228706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.228584    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d50dbe94-4da7-4160-ab95-7f028a88d5b6-sys\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.228706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.228637    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d50dbe94-4da7-4160-ab95-7f028a88d5b6-root\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.228706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.228657    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-wtmp\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.228706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.228678    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-tls\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.228706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.228697    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d50dbe94-4da7-4160-ab95-7f028a88d5b6-root\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.228980 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.228751    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d50dbe94-4da7-4160-ab95-7f028a88d5b6-metrics-client-ca\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.228980 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.228811    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d29b\" (UniqueName: \"kubernetes.io/projected/d50dbe94-4da7-4160-ab95-7f028a88d5b6-kube-api-access-8d29b\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.228980 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.228948    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.229482 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.229462    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d50dbe94-4da7-4160-ab95-7f028a88d5b6-metrics-client-ca\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.230907 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.230884    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-tls\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.231025 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.231006    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d50dbe94-4da7-4160-ab95-7f028a88d5b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.237721 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.237700    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d29b\" (UniqueName: \"kubernetes.io/projected/d50dbe94-4da7-4160-ab95-7f028a88d5b6-kube-api-access-8d29b\") pod \"node-exporter-mfkkg\" (UID: \"d50dbe94-4da7-4160-ab95-7f028a88d5b6\") " pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.330537 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.330503    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mfkkg"
Jul 14 14:32:26.340762 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:32:26.340731    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50dbe94_4da7_4160_ab95_7f028a88d5b6.slice/crio-8cec7e2e4382a4f261adb86da98a34974a20ca0a6e3127d5c61929d5f6b3fc9a WatchSource:0}: Error finding container 8cec7e2e4382a4f261adb86da98a34974a20ca0a6e3127d5c61929d5f6b3fc9a: Status 404 returned error can't find the container with id 8cec7e2e4382a4f261adb86da98a34974a20ca0a6e3127d5c61929d5f6b3fc9a
Jul 14 14:32:26.354717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.354684    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mfkkg" event={"ID":"d50dbe94-4da7-4160-ab95-7f028a88d5b6","Type":"ContainerStarted","Data":"8cec7e2e4382a4f261adb86da98a34974a20ca0a6e3127d5c61929d5f6b3fc9a"}
Jul 14 14:32:26.632925 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.632827    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41817d03-f7be-4b25-991e-7deaaf44b9dc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-vjcnd\" (UID: \"41817d03-f7be-4b25-991e-7deaaf44b9dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:26.633076 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.632955    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.635387 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.635356    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41817d03-f7be-4b25-991e-7deaaf44b9dc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-vjcnd\" (UID: \"41817d03-f7be-4b25-991e-7deaaf44b9dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:26.635387 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.635367    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e84e3d2-b8b1-48e4-81a5-6d5d28d42898-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-56qqr\" (UID: \"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:26.890376 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.890289    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"
Jul 14 14:32:26.896292 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:26.896262    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr"
Jul 14 14:32:27.074238 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:27.074207    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd"]
Jul 14 14:32:27.078652 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:32:27.078620    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41817d03_f7be_4b25_991e_7deaaf44b9dc.slice/crio-92a94f99936783024e20aeafa1c0c4a16e508bea6a26d2c2b4a12706b6490d56 WatchSource:0}: Error finding container 92a94f99936783024e20aeafa1c0c4a16e508bea6a26d2c2b4a12706b6490d56: Status 404 returned error can't find the container with id 92a94f99936783024e20aeafa1c0c4a16e508bea6a26d2c2b4a12706b6490d56
Jul 14 14:32:27.091479 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:27.091442    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-56qqr"]
Jul 14 14:32:27.364772 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:27.364732    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd" event={"ID":"41817d03-f7be-4b25-991e-7deaaf44b9dc","Type":"ContainerStarted","Data":"75699e164099c3422a5661ed3ce7668bf70f38ea09129c835b3614a21258fa6d"}
Jul 14 14:32:27.364919 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:27.364843    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd" event={"ID":"41817d03-f7be-4b25-991e-7deaaf44b9dc","Type":"ContainerStarted","Data":"92a94f99936783024e20aeafa1c0c4a16e508bea6a26d2c2b4a12706b6490d56"}
Jul 14 14:32:27.365896 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:27.365859    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr" event={"ID":"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898","Type":"ContainerStarted","Data":"43a6d65b63f191545b775b7090d923992e63deecc85c2b4f79e13d15666afafa"}
Jul 14 14:32:28.371181 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:28.371145    2586 generic.go:356] "Generic (PLEG): container finished" podID="d50dbe94-4da7-4160-ab95-7f028a88d5b6" containerID="a6e12b7d56232ad47f0ca099d95ab7b792f4e59335ba5fc561553cc3f1cb4a55" exitCode=0
Jul 14 14:32:28.371669 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:28.371241    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mfkkg" event={"ID":"d50dbe94-4da7-4160-ab95-7f028a88d5b6","Type":"ContainerDied","Data":"a6e12b7d56232ad47f0ca099d95ab7b792f4e59335ba5fc561553cc3f1cb4a55"}
Jul 14 14:32:28.373073 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:28.373043    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd" event={"ID":"41817d03-f7be-4b25-991e-7deaaf44b9dc","Type":"ContainerStarted","Data":"5b98dea7193f67fd0ed41bb670e5ec77d56057eda28c493c191fd46ca29baffe"}
Jul 14 14:32:28.970735 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:28.970695    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"]
Jul 14 14:32:28.973470 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:28.973446    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:28.976504 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:28.976430    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 14 14:32:28.976504 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:28.976469    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 14 14:32:28.976504 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:28.976490    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 14 14:32:28.976504 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:28.976500    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m2qbv\""
Jul 14 14:32:28.976793 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:28.976481    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 14 14:32:28.976793 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:28.976430    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 14 14:32:28.976793 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:28.976480    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a5e5vv7jckirh\""
Jul 14 14:32:28.985723 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:28.985696    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"]
Jul 14 14:32:29.061837 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.061804    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.061837 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.061841    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.062107 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.061859    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.062107 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.061887    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-grpc-tls\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.062107 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.061949    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-tls\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.062107 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.062034    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/698fc65b-ac4e-4a74-847d-f23611312764-metrics-client-ca\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.062107 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.062078    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.062285 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.062120    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5dd8\" (UniqueName: \"kubernetes.io/projected/698fc65b-ac4e-4a74-847d-f23611312764-kube-api-access-w5dd8\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.163367 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.163319    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-grpc-tls\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.163367 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.163370    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-tls\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.163637 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.163454    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/698fc65b-ac4e-4a74-847d-f23611312764-metrics-client-ca\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.163637 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.163491    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.163637 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.163541    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5dd8\" (UniqueName: \"kubernetes.io/projected/698fc65b-ac4e-4a74-847d-f23611312764-kube-api-access-w5dd8\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.163637 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.163600    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.163823 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.163654    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.163823 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.163777    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.164484 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.164402    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/698fc65b-ac4e-4a74-847d-f23611312764-metrics-client-ca\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.166133 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.166104    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.166237 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.166108    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-tls\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.166609 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.166587    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-grpc-tls\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.166703 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.166682    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.166703 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.166695    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.166813 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.166775    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/698fc65b-ac4e-4a74-847d-f23611312764-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.172500 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.172473    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5dd8\" (UniqueName: \"kubernetes.io/projected/698fc65b-ac4e-4a74-847d-f23611312764-kube-api-access-w5dd8\") pod \"thanos-querier-797bb6c9fc-4zqqz\" (UID: \"698fc65b-ac4e-4a74-847d-f23611312764\") " pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.282847 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.282751    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:29.381020 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.380975    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr" event={"ID":"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898","Type":"ContainerStarted","Data":"266e17722e5e4e621177bb43607666ed604b61d264b3f6e0022248fde44100e7"}
Jul 14 14:32:29.381516 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.381057    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr" event={"ID":"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898","Type":"ContainerStarted","Data":"551f915b16a14a21570ff4c8df349ca70d268196b07f9c26410f9eae07bf3c3e"}
Jul 14 14:32:29.381516 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.381088    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr" event={"ID":"6e84e3d2-b8b1-48e4-81a5-6d5d28d42898","Type":"ContainerStarted","Data":"08739dce7b411a6830ec1f5e06adc3f1f4b427a6ade6268f3a8c1985fca556bc"}
Jul 14 14:32:29.383401 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.383372    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mfkkg" event={"ID":"d50dbe94-4da7-4160-ab95-7f028a88d5b6","Type":"ContainerStarted","Data":"b704470abdb41f7df1d645401cb388816ee9f251ef3676fe9b2892643a9f060c"}
Jul 14 14:32:29.383541 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.383443    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mfkkg" event={"ID":"d50dbe94-4da7-4160-ab95-7f028a88d5b6","Type":"ContainerStarted","Data":"1d85181e2b7b2423150d8f48677b17c4b96a620a06b4a97690f7393a116f9cc9"}
Jul 14 14:32:29.398436 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.398372    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-56qqr" podStartSLOduration=3.112243877 podStartE2EDuration="4.398354237s" totalImagesPullingTime="1.28611036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:29.397520383 +0000 UTC m=+127.063706799" watchObservedRunningTime="2026-07-14 14:32:29.398354237 +0000 UTC m=+127.064540658"
Jul 14 14:32:29.425782 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.425589    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mfkkg" podStartSLOduration=3.311837459 podStartE2EDuration="4.425565286s" totalImagesPullingTime="1.036410512s" totalInitContainerRuntime="77.317315ms" isStatefulPod=true podCreationTimestamp="2026-07-14 14:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:29.424085429 +0000 UTC m=+127.090271850" watchObservedRunningTime="2026-07-14 14:32:29.425565286 +0000 UTC m=+127.091751708"
Jul 14 14:32:29.431807 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:32:29.431762    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698fc65b_ac4e_4a74_847d_f23611312764.slice/crio-67a1a6223c838ba446b927b3d82101b07fdaefb0ac6b69e302f8a8a7dfd5ec15 WatchSource:0}: Error finding container 67a1a6223c838ba446b927b3d82101b07fdaefb0ac6b69e302f8a8a7dfd5ec15: Status 404 returned error can't find the container with id 67a1a6223c838ba446b927b3d82101b07fdaefb0ac6b69e302f8a8a7dfd5ec15
Jul 14 14:32:29.433710 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:29.432881    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"]
Jul 14 14:32:30.388583 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:30.388546    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd" event={"ID":"41817d03-f7be-4b25-991e-7deaaf44b9dc","Type":"ContainerStarted","Data":"d7faa46f40a2d7d8956f5629b0ad0ddf59b6dfd90021916d56c9f8ba0d284f90"}
Jul 14 14:32:30.389791 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:30.389766    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz" event={"ID":"698fc65b-ac4e-4a74-847d-f23611312764","Type":"ContainerStarted","Data":"67a1a6223c838ba446b927b3d82101b07fdaefb0ac6b69e302f8a8a7dfd5ec15"}
Jul 14 14:32:30.407773 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:30.407720    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-vjcnd" podStartSLOduration=2.978588275 podStartE2EDuration="5.407703727s" totalImagesPullingTime="2.429115452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:30.40716167 +0000 UTC m=+128.073348092" watchObservedRunningTime="2026-07-14 14:32:30.407703727 +0000 UTC m=+128.073890169"
Jul 14 14:32:31.198111 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.198079    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"]
Jul 14 14:32:31.201013 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.200990    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.204633 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.204611    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 14 14:32:31.204764 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.204739    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 14 14:32:31.205056 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.205032    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 14 14:32:31.205229 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.205208    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 14 14:32:31.206298 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.206279    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fl696\""
Jul 14 14:32:31.206402 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.206330    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 14 14:32:31.212372 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.212346    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 14 14:32:31.225268 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.225234    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"]
Jul 14 14:32:31.283930 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.283892    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-federate-client-tls\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.284074 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.283939    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.284074 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.283993    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-telemeter-client-tls\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.284074 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.284016    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.284074 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.284054    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-secret-telemeter-client\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.284214 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.284079    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqfwj\" (UniqueName: \"kubernetes.io/projected/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-kube-api-access-rqfwj\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.284214 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.284095    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-serving-certs-ca-bundle\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.284214 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.284152    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-metrics-client-ca\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.385182 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.385143    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-federate-client-tls\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.385375 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.385204    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.385375 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.385243    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-telemeter-client-tls\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.385375 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.385271    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.385375 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.385315    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-secret-telemeter-client\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.385375 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.385341    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfwj\" (UniqueName: \"kubernetes.io/projected/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-kube-api-access-rqfwj\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.385375 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.385369    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-serving-certs-ca-bundle\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.385696 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.385430    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-metrics-client-ca\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.386207 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.386179    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-metrics-client-ca\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.386550 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.386251    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.386550 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.386331    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-serving-certs-ca-bundle\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.388464 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.388393    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-federate-client-tls\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.388587 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.388461    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-telemeter-client-tls\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.388918 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.388893    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-secret-telemeter-client\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.389277 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.388900    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.395109 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.395082    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfwj\" (UniqueName: \"kubernetes.io/projected/f3e6b198-b9d4-48d2-a3c7-7c4528352d8f-kube-api-access-rqfwj\") pod \"telemeter-client-6df7fcbc56-ptd2m\" (UID: \"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f\") " pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.512112 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.512026    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"
Jul 14 14:32:31.697875 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:31.697840    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m"]
Jul 14 14:32:31.701607 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:32:31.701567    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e6b198_b9d4_48d2_a3c7_7c4528352d8f.slice/crio-322ff4a6d8230225e4fb70a3abf28cbec8d87570c7db2092a24defcc3f88f83d WatchSource:0}: Error finding container 322ff4a6d8230225e4fb70a3abf28cbec8d87570c7db2092a24defcc3f88f83d: Status 404 returned error can't find the container with id 322ff4a6d8230225e4fb70a3abf28cbec8d87570c7db2092a24defcc3f88f83d
Jul 14 14:32:32.229199 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.228945    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 14:32:32.234602 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.234571    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.238391 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.237565    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 14 14:32:32.238391 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.237610    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wljsp\""
Jul 14 14:32:32.238391 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.237634    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 14 14:32:32.238391 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.237657    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 14 14:32:32.238391 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.237614    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e1rm4oa42lj6n\""
Jul 14 14:32:32.238391 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.237576    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 14 14:32:32.238391 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.238018    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 14 14:32:32.238391 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.238032    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 14 14:32:32.238391 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.238083    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 14 14:32:32.238391 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.238218    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 14 14:32:32.239009 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.238773    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 14 14:32:32.239009 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.239000    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 14 14:32:32.239169 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.239147    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 14 14:32:32.239301 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.239153    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 14 14:32:32.239301 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.239239    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 14 14:32:32.242926 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.242880    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 14 14:32:32.252497 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.252470    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 14:32:32.255467 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.255441    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 14 14:32:32.299953 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.299918    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300153 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.299976    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300153 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300006    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j5gd\" (UniqueName: \"kubernetes.io/projected/2dcffa57-99f4-4d95-9afb-13652c884b00-kube-api-access-6j5gd\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300153 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300101    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300301 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300164    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300301 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300226    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300301 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300279    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300450 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300309    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300494 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300444    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2dcffa57-99f4-4d95-9afb-13652c884b00-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300494 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300473    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300592 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300494    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300592 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300575    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2dcffa57-99f4-4d95-9afb-13652c884b00-config-out\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300723 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300606    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300723 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300646    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300723 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300666    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300723 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300682    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-web-config\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300723 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300699    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300940 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300749    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300940 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300786    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-config\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.300940 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.300812    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.400888 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.400848    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m" event={"ID":"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f","Type":"ContainerStarted","Data":"322ff4a6d8230225e4fb70a3abf28cbec8d87570c7db2092a24defcc3f88f83d"}
Jul 14 14:32:32.401351 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.401252    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.401351 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.401295    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.401351 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.401315    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j5gd\" (UniqueName: \"kubernetes.io/projected/2dcffa57-99f4-4d95-9afb-13652c884b00-kube-api-access-6j5gd\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.401351 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.401332    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.401609 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.401468    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.401609 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.401511    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.401609 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.401536    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.401609 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.401560    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.401908 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.401883    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2dcffa57-99f4-4d95-9afb-13652c884b00-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.401908 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.401901    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.402012 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.401926    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.402012 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.401953    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.402356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.402050    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2dcffa57-99f4-4d95-9afb-13652c884b00-config-out\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.402356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.402084    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.402356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.402113    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.402356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.402142    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.402356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.402168    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-web-config\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.402356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.402193    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.402356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.402223    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.402356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.402252    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-config\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.402356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.402288    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.402841 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.402592    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.405405 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.403861    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.405405 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.404906    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.405405 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.405158    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.405405 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.405309    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.405405 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.405367    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2dcffa57-99f4-4d95-9afb-13652c884b00-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.405737 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.405478    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.405737 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.405487    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.405840 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.405764    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.406198 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.406171    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.406287 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.406251    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.406592 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.406564    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.407612 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.407566    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.408373 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.408326    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.408768 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.408723    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2dcffa57-99f4-4d95-9afb-13652c884b00-config-out\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.409002 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.408983    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-web-config\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.409542 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.409521    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.411101 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.411049    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j5gd\" (UniqueName: \"kubernetes.io/projected/2dcffa57-99f4-4d95-9afb-13652c884b00-kube-api-access-6j5gd\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.411593 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.411572    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-config\") pod \"prometheus-k8s-0\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.557561 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.557136    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:32.706637 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.706340    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs\") pod \"network-metrics-daemon-zqd7b\" (UID: \"85b30a97-2ff1-4091-9e77-4359a1c4c242\") " pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:32:32.709823 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.709792    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85b30a97-2ff1-4091-9e77-4359a1c4c242-metrics-certs\") pod \"network-metrics-daemon-zqd7b\" (UID: \"85b30a97-2ff1-4091-9e77-4359a1c4c242\") " pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:32:32.861753 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.861678    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 14:32:32.864359 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:32:32.864331    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dcffa57_99f4_4d95_9afb_13652c884b00.slice/crio-8c44f313b9d232cdfc7d3e7e647860990e7206bcb9eea4a036a4f99e0e39e578 WatchSource:0}: Error finding container 8c44f313b9d232cdfc7d3e7e647860990e7206bcb9eea4a036a4f99e0e39e578: Status 404 returned error can't find the container with id 8c44f313b9d232cdfc7d3e7e647860990e7206bcb9eea4a036a4f99e0e39e578
Jul 14 14:32:32.883527 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.883495    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-shzjn\""
Jul 14 14:32:32.891912 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:32.891885    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqd7b"
Jul 14 14:32:33.048193 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:33.047952    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zqd7b"]
Jul 14 14:32:33.050702 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:32:33.050667    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b30a97_2ff1_4091_9e77_4359a1c4c242.slice/crio-b4252c46a06c193a98b18e335f85c57c0523a9e1e5614d5501e4286668e9619d WatchSource:0}: Error finding container b4252c46a06c193a98b18e335f85c57c0523a9e1e5614d5501e4286668e9619d: Status 404 returned error can't find the container with id b4252c46a06c193a98b18e335f85c57c0523a9e1e5614d5501e4286668e9619d
Jul 14 14:32:33.407283 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:33.407233    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zqd7b" event={"ID":"85b30a97-2ff1-4091-9e77-4359a1c4c242","Type":"ContainerStarted","Data":"b4252c46a06c193a98b18e335f85c57c0523a9e1e5614d5501e4286668e9619d"}
Jul 14 14:32:33.408763 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:33.408730    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerStarted","Data":"8c44f313b9d232cdfc7d3e7e647860990e7206bcb9eea4a036a4f99e0e39e578"}
Jul 14 14:32:33.411491 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:33.411399    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz" event={"ID":"698fc65b-ac4e-4a74-847d-f23611312764","Type":"ContainerStarted","Data":"ddb33bdbe060f295f20627ffcc572d646d207fa05309f91a0f6ff5ed9958928c"}
Jul 14 14:32:33.411601 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:33.411495    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz" event={"ID":"698fc65b-ac4e-4a74-847d-f23611312764","Type":"ContainerStarted","Data":"e97cce1e55ae15312edca3335f55c7d53dd731fd0d09d54e48f7ac61951978d8"}
Jul 14 14:32:33.411601 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:33.411527    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz" event={"ID":"698fc65b-ac4e-4a74-847d-f23611312764","Type":"ContainerStarted","Data":"7626824be723f40fe80d56a2e987de061650e4eda3b3a52ca2dece9602b34c40"}
Jul 14 14:32:35.421901 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:35.421818    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m" event={"ID":"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f","Type":"ContainerStarted","Data":"28306e3fdb817392becdc5f98648e1f2b9b1f52d739ec6909065393c939e621d"}
Jul 14 14:32:35.421901 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:35.421871    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m" event={"ID":"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f","Type":"ContainerStarted","Data":"6f1c170349dcd2dd4d0c1231fd808fa5f40e0ac16fda9f909aa106c4f110147a"}
Jul 14 14:32:35.421901 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:35.421891    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m" event={"ID":"f3e6b198-b9d4-48d2-a3c7-7c4528352d8f","Type":"ContainerStarted","Data":"77321c5fe6945ae4e43e96fa21120ce232d918cba4d119de8008838b59f4b61d"}
Jul 14 14:32:35.423716 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:35.423689    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zqd7b" event={"ID":"85b30a97-2ff1-4091-9e77-4359a1c4c242","Type":"ContainerStarted","Data":"8ff0d5af86ee258ec48e2ee366689eedb305a555827356e3d2b264790ab938d1"}
Jul 14 14:32:35.423835 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:35.423735    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zqd7b" event={"ID":"85b30a97-2ff1-4091-9e77-4359a1c4c242","Type":"ContainerStarted","Data":"df446f5f3b7f37cc1040adf3341371120f1d05f405b852710eaa1a123f22be76"}
Jul 14 14:32:35.425181 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:35.425157    2586 generic.go:356] "Generic (PLEG): container finished" podID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerID="783cda7d020d5ebd789efc2c73c70fc7681f795602957bf4a32fc08a43f2a269" exitCode=0
Jul 14 14:32:35.425264 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:35.425241    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerDied","Data":"783cda7d020d5ebd789efc2c73c70fc7681f795602957bf4a32fc08a43f2a269"}
Jul 14 14:32:35.428343 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:35.428324    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz" event={"ID":"698fc65b-ac4e-4a74-847d-f23611312764","Type":"ContainerStarted","Data":"6338b2ef02ca2994cf2c6e5212a4cad54bfe45ef8ebbef820cff557dc90fabf0"}
Jul 14 14:32:35.428427 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:35.428377    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz" event={"ID":"698fc65b-ac4e-4a74-847d-f23611312764","Type":"ContainerStarted","Data":"2a1c3a00e2a74490f17ef392fbcca68a82540124fbbd4017f6dcfce36be16018"}
Jul 14 14:32:35.428478 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:35.428404    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz" event={"ID":"698fc65b-ac4e-4a74-847d-f23611312764","Type":"ContainerStarted","Data":"e0155a0d9f7b0bcdf0946cdf4be6cb18c81111f7a06aa7f092f2e3553121bb2b"}
Jul 14 14:32:35.428515 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:35.428477    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:35.446562 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:35.446514    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6df7fcbc56-ptd2m" podStartSLOduration=1.402951526 podStartE2EDuration="4.446498124s" totalImagesPullingTime="3.043546598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:35.444186991 +0000 UTC m=+133.110373411" watchObservedRunningTime="2026-07-14 14:32:35.446498124 +0000 UTC m=+133.112684546"
Jul 14 14:32:35.465813 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:35.465765    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz" podStartSLOduration=2.380051322 podStartE2EDuration="7.465749389s" totalImagesPullingTime="5.085698067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:35.46493602 +0000 UTC m=+133.131122440" watchObservedRunningTime="2026-07-14 14:32:35.465749389 +0000 UTC m=+133.131935810"
Jul 14 14:32:35.514053 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:35.513995    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zqd7b" podStartSLOduration=129.764321881 podStartE2EDuration="2m11.513979217s" totalImagesPullingTime="1.749657336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:35.485271271 +0000 UTC m=+133.151457692" watchObservedRunningTime="2026-07-14 14:32:35.513979217 +0000 UTC m=+133.180165634"
Jul 14 14:32:40.456699 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:40.456666    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerStarted","Data":"a3e931edd83707bb243b6e01187965610867b10ef5da872bf7bdd3fd2b156ee6"}
Jul 14 14:32:40.457041 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:40.456766    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerStarted","Data":"7889f2d9bbfbcc20ac9e4d0e93e15ae6ecde50ab1a331fecb4dcf8b9a68d4c7c"}
Jul 14 14:32:40.457041 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:40.456801    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerStarted","Data":"fd729a6a501f8f7e9c7377455315fed6f4c4989bbf1bbd3ea0d46b42dcf489e7"}
Jul 14 14:32:40.457041 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:40.456826    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerStarted","Data":"23cebd009b994f6120ecbdb98af4241cb553a253e2b8283d95c1717f80a9deaa"}
Jul 14 14:32:40.457041 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:40.456849    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerStarted","Data":"10b726795b518123411f6d65cbb78855af0694ac2daa3208b30daaedf21619a4"}
Jul 14 14:32:41.439168 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:41.439139    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-797bb6c9fc-4zqqz"
Jul 14 14:32:41.463137 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:41.463106    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerStarted","Data":"01fa3437013d02257ad63d26d0d286c0cbd04d6a87be7d13348e4a1c9955327e"}
Jul 14 14:32:41.498005 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:41.497944    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.029742481 podStartE2EDuration="9.497923297s" totalImagesPullingTime="6.439328116s" totalInitContainerRuntime="28.8527ms" isStatefulPod=false podCreationTimestamp="2026-07-14 14:32:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:32:41.495290729 +0000 UTC m=+139.161477151" watchObservedRunningTime="2026-07-14 14:32:41.497923297 +0000 UTC m=+139.164109719"
Jul 14 14:32:42.558638 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:42.558594    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:32:54.521431 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:54.521377    2586 generic.go:356] "Generic (PLEG): container finished" podID="a2e3a1f9-cde2-4a79-ae33-9467155e2f14" containerID="a4ba317e56966b57f83fccae0f3387ef85a2cf170113fc59dcca7898c12a42a7" exitCode=0
Jul 14 14:32:54.521841 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:54.521438    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg" event={"ID":"a2e3a1f9-cde2-4a79-ae33-9467155e2f14","Type":"ContainerDied","Data":"a4ba317e56966b57f83fccae0f3387ef85a2cf170113fc59dcca7898c12a42a7"}
Jul 14 14:32:54.521841 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:54.521790    2586 scope.go:117] "RemoveContainer" containerID="a4ba317e56966b57f83fccae0f3387ef85a2cf170113fc59dcca7898c12a42a7"
Jul 14 14:32:55.527353 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:32:55.527316    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-hqvkg" event={"ID":"a2e3a1f9-cde2-4a79-ae33-9467155e2f14","Type":"ContainerStarted","Data":"30b0d4582a07c046e6a5681f3c3b2b37c663b63371ae26d7c23ac44b6b587b51"}
Jul 14 14:33:32.558334 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:32.558248    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:32.574644 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:32.574612    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:32.696706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:32.696677    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:50.578889 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.578854    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 14:33:50.579386 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.579189    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="prometheus" containerID="cri-o://10b726795b518123411f6d65cbb78855af0694ac2daa3208b30daaedf21619a4" gracePeriod=600
Jul 14 14:33:50.579386 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.579205    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="kube-rbac-proxy" containerID="cri-o://a3e931edd83707bb243b6e01187965610867b10ef5da872bf7bdd3fd2b156ee6" gracePeriod=600
Jul 14 14:33:50.579386 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.579247    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="thanos-sidecar" containerID="cri-o://fd729a6a501f8f7e9c7377455315fed6f4c4989bbf1bbd3ea0d46b42dcf489e7" gracePeriod=600
Jul 14 14:33:50.579386 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.579270    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="kube-rbac-proxy-thanos" containerID="cri-o://01fa3437013d02257ad63d26d0d286c0cbd04d6a87be7d13348e4a1c9955327e" gracePeriod=600
Jul 14 14:33:50.579386 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.579321    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="kube-rbac-proxy-web" containerID="cri-o://7889f2d9bbfbcc20ac9e4d0e93e15ae6ecde50ab1a331fecb4dcf8b9a68d4c7c" gracePeriod=600
Jul 14 14:33:50.579386 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.579271    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="config-reloader" containerID="cri-o://23cebd009b994f6120ecbdb98af4241cb553a253e2b8283d95c1717f80a9deaa" gracePeriod=600
Jul 14 14:33:50.754580 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.754547    2586 generic.go:356] "Generic (PLEG): container finished" podID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerID="01fa3437013d02257ad63d26d0d286c0cbd04d6a87be7d13348e4a1c9955327e" exitCode=0
Jul 14 14:33:50.754580 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.754580    2586 generic.go:356] "Generic (PLEG): container finished" podID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerID="a3e931edd83707bb243b6e01187965610867b10ef5da872bf7bdd3fd2b156ee6" exitCode=0
Jul 14 14:33:50.754875 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.754590    2586 generic.go:356] "Generic (PLEG): container finished" podID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerID="7889f2d9bbfbcc20ac9e4d0e93e15ae6ecde50ab1a331fecb4dcf8b9a68d4c7c" exitCode=0
Jul 14 14:33:50.754875 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.754597    2586 generic.go:356] "Generic (PLEG): container finished" podID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerID="fd729a6a501f8f7e9c7377455315fed6f4c4989bbf1bbd3ea0d46b42dcf489e7" exitCode=0
Jul 14 14:33:50.754875 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.754604    2586 generic.go:356] "Generic (PLEG): container finished" podID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerID="23cebd009b994f6120ecbdb98af4241cb553a253e2b8283d95c1717f80a9deaa" exitCode=0
Jul 14 14:33:50.754875 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.754611    2586 generic.go:356] "Generic (PLEG): container finished" podID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerID="10b726795b518123411f6d65cbb78855af0694ac2daa3208b30daaedf21619a4" exitCode=0
Jul 14 14:33:50.754875 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.754723    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerDied","Data":"01fa3437013d02257ad63d26d0d286c0cbd04d6a87be7d13348e4a1c9955327e"}
Jul 14 14:33:50.754875 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.754796    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerDied","Data":"a3e931edd83707bb243b6e01187965610867b10ef5da872bf7bdd3fd2b156ee6"}
Jul 14 14:33:50.754875 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.754837    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerDied","Data":"7889f2d9bbfbcc20ac9e4d0e93e15ae6ecde50ab1a331fecb4dcf8b9a68d4c7c"}
Jul 14 14:33:50.754875 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.754862    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerDied","Data":"fd729a6a501f8f7e9c7377455315fed6f4c4989bbf1bbd3ea0d46b42dcf489e7"}
Jul 14 14:33:50.755206 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.754885    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerDied","Data":"23cebd009b994f6120ecbdb98af4241cb553a253e2b8283d95c1717f80a9deaa"}
Jul 14 14:33:50.755206 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.754922    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerDied","Data":"10b726795b518123411f6d65cbb78855af0694ac2daa3208b30daaedf21619a4"}
Jul 14 14:33:50.833013 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.832951    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:50.918131 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918098    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-grpc-tls\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918317 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918148    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-1\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918317 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918182    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-tls\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918317 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918215    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-trusted-ca-bundle\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918317 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918248    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2dcffa57-99f4-4d95-9afb-13652c884b00-tls-assets\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918317 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918274    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-serving-certs-ca-bundle\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918661 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918331    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-kubelet-serving-ca-bundle\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918661 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918356    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-metrics-client-ca\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918661 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918435    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918661 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918467    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-config\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918661 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918500    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-thanos-prometheus-http-client-file\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918661 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918529    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-db\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918661 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918552    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-web-config\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918661 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918581    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-metrics-client-certs\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918661 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918621    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.918661 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918629    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:33:50.918661 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918644    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-2\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.919146 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918656    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:33:50.919146 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918712    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j5gd\" (UniqueName: \"kubernetes.io/projected/2dcffa57-99f4-4d95-9afb-13652c884b00-kube-api-access-6j5gd\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.919146 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918742    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-0\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.919146 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918780    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2dcffa57-99f4-4d95-9afb-13652c884b00-config-out\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.919146 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.918816    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-kube-rbac-proxy\") pod \"2dcffa57-99f4-4d95-9afb-13652c884b00\" (UID: \"2dcffa57-99f4-4d95-9afb-13652c884b00\") "
Jul 14 14:33:50.919769 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.919503    2586 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:50.919769 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.919530    2586 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:50.919769 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.919665    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:33:50.919769 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.919688    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:33:50.920060 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.919795    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:33:50.920060 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.920035    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:33:50.920171 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.920117    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:33:50.921020 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.920857    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:33:50.921435 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.921392    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcffa57-99f4-4d95-9afb-13652c884b00-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:33:50.922201 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.922169    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:33:50.924075 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.923973    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:33:50.924075 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.923987    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:33:50.924075 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.924040    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:33:50.924388 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.924356    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcffa57-99f4-4d95-9afb-13652c884b00-kube-api-access-6j5gd" (OuterVolumeSpecName: "kube-api-access-6j5gd") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "kube-api-access-6j5gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:33:50.924575 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.924531    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:33:50.924799 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.924770    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcffa57-99f4-4d95-9afb-13652c884b00-config-out" (OuterVolumeSpecName: "config-out") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:33:50.924911 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.924891    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:33:50.925159 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.925140    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-config" (OuterVolumeSpecName: "config") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:33:50.925545 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.925528    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:33:50.934780 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:50.934759    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-web-config" (OuterVolumeSpecName: "web-config") pod "2dcffa57-99f4-4d95-9afb-13652c884b00" (UID: "2dcffa57-99f4-4d95-9afb-13652c884b00"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:33:51.020526 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020483    2586 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020526 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020522    2586 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-db\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020526 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020533    2586 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-web-config\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020542    2586 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-metrics-client-certs\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020552    2586 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020561    2586 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020570    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j5gd\" (UniqueName: \"kubernetes.io/projected/2dcffa57-99f4-4d95-9afb-13652c884b00-kube-api-access-6j5gd\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020578    2586 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020589    2586 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2dcffa57-99f4-4d95-9afb-13652c884b00-config-out\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020602    2586 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-kube-rbac-proxy\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020612    2586 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-grpc-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020620    2586 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020628    2586 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2dcffa57-99f4-4d95-9afb-13652c884b00-tls-assets\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020636    2586 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020644    2586 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020652    2586 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dcffa57-99f4-4d95-9afb-13652c884b00-configmap-metrics-client-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020660    2586 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.020756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.020670    2586 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2dcffa57-99f4-4d95-9afb-13652c884b00-config\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:33:51.761506 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.761474    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2dcffa57-99f4-4d95-9afb-13652c884b00","Type":"ContainerDied","Data":"8c44f313b9d232cdfc7d3e7e647860990e7206bcb9eea4a036a4f99e0e39e578"}
Jul 14 14:33:51.761863 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.761534    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.761863 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.761551    2586 scope.go:117] "RemoveContainer" containerID="01fa3437013d02257ad63d26d0d286c0cbd04d6a87be7d13348e4a1c9955327e"
Jul 14 14:33:51.769851 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.769833    2586 scope.go:117] "RemoveContainer" containerID="a3e931edd83707bb243b6e01187965610867b10ef5da872bf7bdd3fd2b156ee6"
Jul 14 14:33:51.777046 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.777024    2586 scope.go:117] "RemoveContainer" containerID="7889f2d9bbfbcc20ac9e4d0e93e15ae6ecde50ab1a331fecb4dcf8b9a68d4c7c"
Jul 14 14:33:51.784550 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.784525    2586 scope.go:117] "RemoveContainer" containerID="fd729a6a501f8f7e9c7377455315fed6f4c4989bbf1bbd3ea0d46b42dcf489e7"
Jul 14 14:33:51.786700 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.786676    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 14:33:51.790408 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.790387    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 14:33:51.792652 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.792637    2586 scope.go:117] "RemoveContainer" containerID="23cebd009b994f6120ecbdb98af4241cb553a253e2b8283d95c1717f80a9deaa"
Jul 14 14:33:51.799561 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.799544    2586 scope.go:117] "RemoveContainer" containerID="10b726795b518123411f6d65cbb78855af0694ac2daa3208b30daaedf21619a4"
Jul 14 14:33:51.807233 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.807212    2586 scope.go:117] "RemoveContainer" containerID="783cda7d020d5ebd789efc2c73c70fc7681f795602957bf4a32fc08a43f2a269"
Jul 14 14:33:51.815764 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.815744    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 14:33:51.816042 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816030    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="kube-rbac-proxy"
Jul 14 14:33:51.816082 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816043    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="kube-rbac-proxy"
Jul 14 14:33:51.816082 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816052    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="init-config-reloader"
Jul 14 14:33:51.816082 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816058    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="init-config-reloader"
Jul 14 14:33:51.816082 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816066    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="thanos-sidecar"
Jul 14 14:33:51.816082 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816071    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="thanos-sidecar"
Jul 14 14:33:51.816082 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816076    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="kube-rbac-proxy-web"
Jul 14 14:33:51.816082 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816081    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="kube-rbac-proxy-web"
Jul 14 14:33:51.816303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816094    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="kube-rbac-proxy-thanos"
Jul 14 14:33:51.816303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816099    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="kube-rbac-proxy-thanos"
Jul 14 14:33:51.816303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816109    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="prometheus"
Jul 14 14:33:51.816303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816114    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="prometheus"
Jul 14 14:33:51.816303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816121    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="config-reloader"
Jul 14 14:33:51.816303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816125    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="config-reloader"
Jul 14 14:33:51.816303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816173    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="thanos-sidecar"
Jul 14 14:33:51.816303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816182    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="prometheus"
Jul 14 14:33:51.816303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816188    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="config-reloader"
Jul 14 14:33:51.816303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816194    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="kube-rbac-proxy-thanos"
Jul 14 14:33:51.816303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816200    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="kube-rbac-proxy-web"
Jul 14 14:33:51.816303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.816206    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" containerName="kube-rbac-proxy"
Jul 14 14:33:51.822556 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.821266    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.825140 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.825114    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 14 14:33:51.825289 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.825270    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e1rm4oa42lj6n\""
Jul 14 14:33:51.825536 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.825522    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 14 14:33:51.825620 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.825530    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 14 14:33:51.826706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.826629    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 14 14:33:51.826706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.826639    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 14 14:33:51.826706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.826632    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wljsp\""
Jul 14 14:33:51.826706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.826691    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 14 14:33:51.826967 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.826636    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 14 14:33:51.826967 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.826871    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 14 14:33:51.826967 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.826944    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 14 14:33:51.827113 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.826946    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 14 14:33:51.827296 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.827255    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 14 14:33:51.827502 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.827472    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 14 14:33:51.827502 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.827474    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 14 14:33:51.830097 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.830078    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 14 14:33:51.832560 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.832539    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 14 14:33:51.838261 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.838241    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 14:33:51.927566 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.927535    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/56c8841f-9fb2-4ece-a4fb-06c5f159491e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.927734 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.927589    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.927734 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.927613    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.927826 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.927743    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.927826 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.927797    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.927891 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.927825    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-web-config\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.927891 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.927853    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.927950 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.927902    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.927979 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.927950    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/56c8841f-9fb2-4ece-a4fb-06c5f159491e-config-out\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.928011 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.927980    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.928040 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.928006    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.928074 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.928049    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.928110 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.928079    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-config\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.928110 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.928105    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.928204 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.928187    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.928249 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.928219    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.928340 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.928244    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x74g8\" (UniqueName: \"kubernetes.io/projected/56c8841f-9fb2-4ece-a4fb-06c5f159491e-kube-api-access-x74g8\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.928340 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.928289    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.928340 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.928319    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:51.928487 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:51.928346    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.028893 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.028855    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.028893 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.028892    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/56c8841f-9fb2-4ece-a4fb-06c5f159491e-config-out\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029083 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.028911    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029083 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029040    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029083 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029071    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029207 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029095    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-config\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029207 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029132    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029557 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029524    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029690 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029559    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029690 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029582    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74g8\" (UniqueName: \"kubernetes.io/projected/56c8841f-9fb2-4ece-a4fb-06c5f159491e-kube-api-access-x74g8\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029690 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029628    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029690 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029651    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029690 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029671    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029930 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029700    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/56c8841f-9fb2-4ece-a4fb-06c5f159491e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029930 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029743    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029930 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029781    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029930 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029815    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029930 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029863    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029930 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029874    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.029930 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029887    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-web-config\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.030269 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029939    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.031623 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.030830    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.031623 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.030861    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.031623 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.031469    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.033547 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.032143    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/56c8841f-9fb2-4ece-a4fb-06c5f159491e-config-out\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.033547 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.032184    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.033547 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.029445    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.033547 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.032274    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.033547 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.032320    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-config\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.033547 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.032465    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.033547 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.032739    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.033547 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.031469    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.033547 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.032828    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.034067 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.034043    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-web-config\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.034724 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.034696    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.035210 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.035184    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.035380 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.035359    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/56c8841f-9fb2-4ece-a4fb-06c5f159491e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.035748 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.035728    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/56c8841f-9fb2-4ece-a4fb-06c5f159491e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.035912 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.035898    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/56c8841f-9fb2-4ece-a4fb-06c5f159491e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.040226 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.040209    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x74g8\" (UniqueName: \"kubernetes.io/projected/56c8841f-9fb2-4ece-a4fb-06c5f159491e-kube-api-access-x74g8\") pod \"prometheus-k8s-0\" (UID: \"56c8841f-9fb2-4ece-a4fb-06c5f159491e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.133464 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.133403    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:33:52.275116 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.275059    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 14:33:52.277692 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:33:52.277657    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c8841f_9fb2_4ece_a4fb_06c5f159491e.slice/crio-1db3d3e49311c6426ed242d98829064aacc42270036f5d87a3cf4169202ab94d WatchSource:0}: Error finding container 1db3d3e49311c6426ed242d98829064aacc42270036f5d87a3cf4169202ab94d: Status 404 returned error can't find the container with id 1db3d3e49311c6426ed242d98829064aacc42270036f5d87a3cf4169202ab94d
Jul 14 14:33:52.769473 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.769437    2586 generic.go:356] "Generic (PLEG): container finished" podID="56c8841f-9fb2-4ece-a4fb-06c5f159491e" containerID="5ee55af0d65edb60225e63b92f4125ff13ab1ab7dac1a794fa55f9187d3de123" exitCode=0
Jul 14 14:33:52.769936 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.769513    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56c8841f-9fb2-4ece-a4fb-06c5f159491e","Type":"ContainerDied","Data":"5ee55af0d65edb60225e63b92f4125ff13ab1ab7dac1a794fa55f9187d3de123"}
Jul 14 14:33:52.769936 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.769600    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56c8841f-9fb2-4ece-a4fb-06c5f159491e","Type":"ContainerStarted","Data":"1db3d3e49311c6426ed242d98829064aacc42270036f5d87a3cf4169202ab94d"}
Jul 14 14:33:52.877630 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:52.877600    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dcffa57-99f4-4d95-9afb-13652c884b00" path="/var/lib/kubelet/pods/2dcffa57-99f4-4d95-9afb-13652c884b00/volumes"
Jul 14 14:33:53.777262 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:53.777222    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56c8841f-9fb2-4ece-a4fb-06c5f159491e","Type":"ContainerStarted","Data":"7c305f8f10d267e04561d58f279c314a02586b5a01cd298de7c766a804a91741"}
Jul 14 14:33:53.777706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:53.777318    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56c8841f-9fb2-4ece-a4fb-06c5f159491e","Type":"ContainerStarted","Data":"7317185969eef659aacfa8bcfb9f6b45f117c49e65ce67c410c3258993999c35"}
Jul 14 14:33:53.777706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:53.777347    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56c8841f-9fb2-4ece-a4fb-06c5f159491e","Type":"ContainerStarted","Data":"ddd6fe77bdb10827105356c702166068b16cb8e1395608760814a99463bc7194"}
Jul 14 14:33:53.777706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:53.777372    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56c8841f-9fb2-4ece-a4fb-06c5f159491e","Type":"ContainerStarted","Data":"2dc5e44fa24c6e771910a86c230abe6fd9fd49401a834a83ae4b5b7ac4f3d82a"}
Jul 14 14:33:53.777706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:53.777398    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56c8841f-9fb2-4ece-a4fb-06c5f159491e","Type":"ContainerStarted","Data":"056e2d8c2902834b1e113bdef828ced31644cd596024b6be9ae1a07f9fdf4581"}
Jul 14 14:33:53.777706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:53.777447    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56c8841f-9fb2-4ece-a4fb-06c5f159491e","Type":"ContainerStarted","Data":"77be3f4e81d253a1b9c5bb2e18c54e6aac2398338fd367d94590141863ce5ad2"}
Jul 14 14:33:53.811354 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:53.811284    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7901064780000002 podStartE2EDuration="2.811262779s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.156301ms" isStatefulPod=false podCreationTimestamp="2026-07-14 14:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:33:53.809394214 +0000 UTC m=+211.475580667" watchObservedRunningTime="2026-07-14 14:33:53.811262779 +0000 UTC m=+211.477449201"
Jul 14 14:33:57.133697 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:33:57.133650    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:34:52.134252 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:34:52.134164    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:34:52.150599 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:34:52.150575    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:34:53.033270 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:34:53.033241    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 14:35:22.768749 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:22.768717    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:35:22.772454 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:22.772409    2586 kubelet.go:1648] "Image garbage collection succeeded"
Jul 14 14:35:22.780802 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:22.780779    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:35:32.597954 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.597916    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"]
Jul 14 14:35:32.601194 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.601177    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:32.604481 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.604448    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 14 14:35:32.604803 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.604776    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 14 14:35:32.605785 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.605767    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 14 14:35:32.605905 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.605889    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 14 14:35:32.605977 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.605962    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vz8fw\""
Jul 14 14:35:32.606021 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.606002    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 14 14:35:32.614042 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.614016    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"]
Jul 14 14:35:32.709001 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.708953    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5e1ad1f2-2d10-4009-a3e9-307393af3049-manager-config\") pod \"lws-controller-manager-fc6b4b856-v6ghr\" (UID: \"5e1ad1f2-2d10-4009-a3e9-307393af3049\") " pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:32.709001 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.709016    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e1ad1f2-2d10-4009-a3e9-307393af3049-cert\") pod \"lws-controller-manager-fc6b4b856-v6ghr\" (UID: \"5e1ad1f2-2d10-4009-a3e9-307393af3049\") " pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:32.709253 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.709054    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5s5r\" (UniqueName: \"kubernetes.io/projected/5e1ad1f2-2d10-4009-a3e9-307393af3049-kube-api-access-p5s5r\") pod \"lws-controller-manager-fc6b4b856-v6ghr\" (UID: \"5e1ad1f2-2d10-4009-a3e9-307393af3049\") " pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:32.709253 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.709108    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e1ad1f2-2d10-4009-a3e9-307393af3049-metrics-cert\") pod \"lws-controller-manager-fc6b4b856-v6ghr\" (UID: \"5e1ad1f2-2d10-4009-a3e9-307393af3049\") " pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:32.809898 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.809861    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5e1ad1f2-2d10-4009-a3e9-307393af3049-manager-config\") pod \"lws-controller-manager-fc6b4b856-v6ghr\" (UID: \"5e1ad1f2-2d10-4009-a3e9-307393af3049\") " pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:32.809898 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.809907    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e1ad1f2-2d10-4009-a3e9-307393af3049-cert\") pod \"lws-controller-manager-fc6b4b856-v6ghr\" (UID: \"5e1ad1f2-2d10-4009-a3e9-307393af3049\") " pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:32.810162 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.809926    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5s5r\" (UniqueName: \"kubernetes.io/projected/5e1ad1f2-2d10-4009-a3e9-307393af3049-kube-api-access-p5s5r\") pod \"lws-controller-manager-fc6b4b856-v6ghr\" (UID: \"5e1ad1f2-2d10-4009-a3e9-307393af3049\") " pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:32.810162 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.809948    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e1ad1f2-2d10-4009-a3e9-307393af3049-metrics-cert\") pod \"lws-controller-manager-fc6b4b856-v6ghr\" (UID: \"5e1ad1f2-2d10-4009-a3e9-307393af3049\") " pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:32.810647 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.810629    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5e1ad1f2-2d10-4009-a3e9-307393af3049-manager-config\") pod \"lws-controller-manager-fc6b4b856-v6ghr\" (UID: \"5e1ad1f2-2d10-4009-a3e9-307393af3049\") " pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:32.812542 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.812516    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e1ad1f2-2d10-4009-a3e9-307393af3049-cert\") pod \"lws-controller-manager-fc6b4b856-v6ghr\" (UID: \"5e1ad1f2-2d10-4009-a3e9-307393af3049\") " pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:32.812645 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.812582    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e1ad1f2-2d10-4009-a3e9-307393af3049-metrics-cert\") pod \"lws-controller-manager-fc6b4b856-v6ghr\" (UID: \"5e1ad1f2-2d10-4009-a3e9-307393af3049\") " pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:32.826685 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.826660    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5s5r\" (UniqueName: \"kubernetes.io/projected/5e1ad1f2-2d10-4009-a3e9-307393af3049-kube-api-access-p5s5r\") pod \"lws-controller-manager-fc6b4b856-v6ghr\" (UID: \"5e1ad1f2-2d10-4009-a3e9-307393af3049\") " pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:32.911036 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:32.910950    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:33.065946 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.065925    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"]
Jul 14 14:35:33.068256 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:35:33.068230    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1ad1f2_2d10_4009_a3e9_307393af3049.slice/crio-95e1ad90ba4dc3f06429c613760f71bd01cff229041068088a9a808c428dcb82 WatchSource:0}: Error finding container 95e1ad90ba4dc3f06429c613760f71bd01cff229041068088a9a808c428dcb82: Status 404 returned error can't find the container with id 95e1ad90ba4dc3f06429c613760f71bd01cff229041068088a9a808c428dcb82
Jul 14 14:35:33.070170 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.070155    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 14:35:33.181858 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.181823    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr" event={"ID":"5e1ad1f2-2d10-4009-a3e9-307393af3049","Type":"ContainerStarted","Data":"95e1ad90ba4dc3f06429c613760f71bd01cff229041068088a9a808c428dcb82"}
Jul 14 14:35:33.830041 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.829835    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"]
Jul 14 14:35:33.834058 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.834037    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:33.838439 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.838388    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 14 14:35:33.838567 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.838508    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 14 14:35:33.838706 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.838679    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-62jlr\""
Jul 14 14:35:33.838788 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.838711    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 14 14:35:33.838788 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.838679    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 14 14:35:33.856936 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.856894    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"]
Jul 14 14:35:33.919809 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.919764    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf559f9d-f954-4dd8-bb53-718ecb7640fb-manifests\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:33.920014 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.919822    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbgp8\" (UniqueName: \"kubernetes.io/projected/cf559f9d-f954-4dd8-bb53-718ecb7640fb-kube-api-access-xbgp8\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:33.920014 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.919896    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf559f9d-f954-4dd8-bb53-718ecb7640fb-tmp\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:33.920014 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.919934    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf559f9d-f954-4dd8-bb53-718ecb7640fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:33.920188 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:33.920019    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf559f9d-f954-4dd8-bb53-718ecb7640fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:34.021165 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:34.021135    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf559f9d-f954-4dd8-bb53-718ecb7640fb-manifests\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:34.021165 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:34.021174    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbgp8\" (UniqueName: \"kubernetes.io/projected/cf559f9d-f954-4dd8-bb53-718ecb7640fb-kube-api-access-xbgp8\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:34.021456 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:34.021371    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf559f9d-f954-4dd8-bb53-718ecb7640fb-tmp\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:34.021456 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:34.021438    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf559f9d-f954-4dd8-bb53-718ecb7640fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:34.021583 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:34.021507    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf559f9d-f954-4dd8-bb53-718ecb7640fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:34.021659 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:34.021637    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf559f9d-f954-4dd8-bb53-718ecb7640fb-manifests\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:34.024044 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:34.024022    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf559f9d-f954-4dd8-bb53-718ecb7640fb-tmp\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:34.024303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:34.024278    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf559f9d-f954-4dd8-bb53-718ecb7640fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:34.024446 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:34.024400    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf559f9d-f954-4dd8-bb53-718ecb7640fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:34.035164 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:34.035131    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbgp8\" (UniqueName: \"kubernetes.io/projected/cf559f9d-f954-4dd8-bb53-718ecb7640fb-kube-api-access-xbgp8\") pod \"opendatahub-operator-controller-manager-b4469fb8c-8qbxj\" (UID: \"cf559f9d-f954-4dd8-bb53-718ecb7640fb\") " pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:34.149562 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:34.149491    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:34.308133 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:34.308101    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"]
Jul 14 14:35:34.311537 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:35:34.311503    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf559f9d_f954_4dd8_bb53_718ecb7640fb.slice/crio-23d4a5a098ce0a6887d1d2866da3730026b1a3766f71eec1f8d086f621528ca0 WatchSource:0}: Error finding container 23d4a5a098ce0a6887d1d2866da3730026b1a3766f71eec1f8d086f621528ca0: Status 404 returned error can't find the container with id 23d4a5a098ce0a6887d1d2866da3730026b1a3766f71eec1f8d086f621528ca0
Jul 14 14:35:35.192576 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:35.192533    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj" event={"ID":"cf559f9d-f954-4dd8-bb53-718ecb7640fb","Type":"ContainerStarted","Data":"23d4a5a098ce0a6887d1d2866da3730026b1a3766f71eec1f8d086f621528ca0"}
Jul 14 14:35:36.200828 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:36.200724    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr" event={"ID":"5e1ad1f2-2d10-4009-a3e9-307393af3049","Type":"ContainerStarted","Data":"00eae95ad3ea6a1d02e00f9280fc23b2834e64a2e79ffd5a71710e1c3b9ca0ae"}
Jul 14 14:35:36.200828 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:36.200814    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:39.216008 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:39.215966    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj" event={"ID":"cf559f9d-f954-4dd8-bb53-718ecb7640fb","Type":"ContainerStarted","Data":"aaeeea96e8761b51148cb4b3203c5d8801ca7c8f9291294b87c2ad201c1d2548"}
Jul 14 14:35:39.246552 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:39.246497    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr" podStartSLOduration=4.620319803 podStartE2EDuration="7.246482801s" totalImagesPullingTime="2.626162998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:35:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:35:36.250955467 +0000 UTC m=+313.917141891" watchObservedRunningTime="2026-07-14 14:35:39.246482801 +0000 UTC m=+316.912669221"
Jul 14 14:35:40.221902 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:40.221871    2586 generic.go:356] "Generic (PLEG): container finished" podID="cf559f9d-f954-4dd8-bb53-718ecb7640fb" containerID="aaeeea96e8761b51148cb4b3203c5d8801ca7c8f9291294b87c2ad201c1d2548" exitCode=0
Jul 14 14:35:40.222296 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:40.221946    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj" event={"ID":"cf559f9d-f954-4dd8-bb53-718ecb7640fb","Type":"ContainerDied","Data":"aaeeea96e8761b51148cb4b3203c5d8801ca7c8f9291294b87c2ad201c1d2548"}
Jul 14 14:35:41.227743 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:41.227709    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj" event={"ID":"cf559f9d-f954-4dd8-bb53-718ecb7640fb","Type":"ContainerStarted","Data":"9bc03db0aa77026057900841ee0dd42eb1a1afbdab7d67e140b9b25cfb3bac38"}
Jul 14 14:35:41.228154 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:41.227827    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:41.251284 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:41.251226    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj" podStartSLOduration=2.934045632 podStartE2EDuration="8.251205561s" totalImagesPullingTime="4.25253395s" totalInitContainerRuntime="1.064625979s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:35:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:35:41.249509315 +0000 UTC m=+318.915695735" watchObservedRunningTime="2026-07-14 14:35:41.251205561 +0000 UTC m=+318.917391985"
Jul 14 14:35:47.208208 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:47.208181    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-fc6b4b856-v6ghr"
Jul 14 14:35:52.234913 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:52.234864    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b4469fb8c-8qbxj"
Jul 14 14:35:54.609818 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:54.609784    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f65956896-97cm7"]
Jul 14 14:35:54.674069 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:54.674036    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f65956896-97cm7"]
Jul 14 14:35:54.674225 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:54.674159    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7"
Jul 14 14:35:54.680136 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:54.680114    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 14 14:35:54.680474 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:54.680453    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pswpq\""
Jul 14 14:35:54.680696 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:54.680674    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 14 14:35:54.707112 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:54.707075    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1782e59-c729-4ce9-a67e-d5f5a9950ad0-tmp\") pod \"kube-auth-proxy-6f65956896-97cm7\" (UID: \"e1782e59-c729-4ce9-a67e-d5f5a9950ad0\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7"
Jul 14 14:35:54.707112 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:54.707111    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw7ns\" (UniqueName: \"kubernetes.io/projected/e1782e59-c729-4ce9-a67e-d5f5a9950ad0-kube-api-access-xw7ns\") pod \"kube-auth-proxy-6f65956896-97cm7\" (UID: \"e1782e59-c729-4ce9-a67e-d5f5a9950ad0\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7"
Jul 14 14:35:54.707300 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:54.707261    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1782e59-c729-4ce9-a67e-d5f5a9950ad0-tls-certs\") pod \"kube-auth-proxy-6f65956896-97cm7\" (UID: \"e1782e59-c729-4ce9-a67e-d5f5a9950ad0\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7"
Jul 14 14:35:54.808657 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:54.808619    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1782e59-c729-4ce9-a67e-d5f5a9950ad0-tls-certs\") pod \"kube-auth-proxy-6f65956896-97cm7\" (UID: \"e1782e59-c729-4ce9-a67e-d5f5a9950ad0\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7"
Jul 14 14:35:54.808831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:54.808675    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1782e59-c729-4ce9-a67e-d5f5a9950ad0-tmp\") pod \"kube-auth-proxy-6f65956896-97cm7\" (UID: \"e1782e59-c729-4ce9-a67e-d5f5a9950ad0\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7"
Jul 14 14:35:54.808831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:54.808704    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw7ns\" (UniqueName: \"kubernetes.io/projected/e1782e59-c729-4ce9-a67e-d5f5a9950ad0-kube-api-access-xw7ns\") pod \"kube-auth-proxy-6f65956896-97cm7\" (UID: \"e1782e59-c729-4ce9-a67e-d5f5a9950ad0\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7"
Jul 14 14:35:54.808831 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:35:54.808785    2586 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 14 14:35:54.808964 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:35:54.808866    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1782e59-c729-4ce9-a67e-d5f5a9950ad0-tls-certs podName:e1782e59-c729-4ce9-a67e-d5f5a9950ad0 nodeName:}" failed. No retries permitted until 2026-07-14 14:35:55.30884605 +0000 UTC m=+332.975032450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e1782e59-c729-4ce9-a67e-d5f5a9950ad0-tls-certs") pod "kube-auth-proxy-6f65956896-97cm7" (UID: "e1782e59-c729-4ce9-a67e-d5f5a9950ad0") : secret "kube-auth-proxy-tls" not found
Jul 14 14:35:54.811566 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:54.811517    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1782e59-c729-4ce9-a67e-d5f5a9950ad0-tmp\") pod \"kube-auth-proxy-6f65956896-97cm7\" (UID: \"e1782e59-c729-4ce9-a67e-d5f5a9950ad0\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7"
Jul 14 14:35:54.833245 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:54.833216    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw7ns\" (UniqueName: \"kubernetes.io/projected/e1782e59-c729-4ce9-a67e-d5f5a9950ad0-kube-api-access-xw7ns\") pod \"kube-auth-proxy-6f65956896-97cm7\" (UID: \"e1782e59-c729-4ce9-a67e-d5f5a9950ad0\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7"
Jul 14 14:35:55.313049 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:55.313009    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1782e59-c729-4ce9-a67e-d5f5a9950ad0-tls-certs\") pod \"kube-auth-proxy-6f65956896-97cm7\" (UID: \"e1782e59-c729-4ce9-a67e-d5f5a9950ad0\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7"
Jul 14 14:35:55.315602 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:55.315581    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1782e59-c729-4ce9-a67e-d5f5a9950ad0-tls-certs\") pod \"kube-auth-proxy-6f65956896-97cm7\" (UID: \"e1782e59-c729-4ce9-a67e-d5f5a9950ad0\") " pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7"
Jul 14 14:35:55.584251 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:55.584157    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7"
Jul 14 14:35:55.796242 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:55.796217    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f65956896-97cm7"]
Jul 14 14:35:55.798342 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:35:55.798310    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1782e59_c729_4ce9_a67e_d5f5a9950ad0.slice/crio-fb9d5432a49b80a0f568d88fe91abcf24a166b5f9fda015738f6373ae274136f WatchSource:0}: Error finding container fb9d5432a49b80a0f568d88fe91abcf24a166b5f9fda015738f6373ae274136f: Status 404 returned error can't find the container with id fb9d5432a49b80a0f568d88fe91abcf24a166b5f9fda015738f6373ae274136f
Jul 14 14:35:56.295829 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:56.295794    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7" event={"ID":"e1782e59-c729-4ce9-a67e-d5f5a9950ad0","Type":"ContainerStarted","Data":"fb9d5432a49b80a0f568d88fe91abcf24a166b5f9fda015738f6373ae274136f"}
Jul 14 14:35:59.311727 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:59.311695    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7" event={"ID":"e1782e59-c729-4ce9-a67e-d5f5a9950ad0","Type":"ContainerStarted","Data":"12e6cfe071a47da81c40c1ed59cd227a4888134cd4152e2aef998240492bcadc"}
Jul 14 14:35:59.331159 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:35:59.331103    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f65956896-97cm7" podStartSLOduration=1.926891242 podStartE2EDuration="5.33108547s" totalImagesPullingTime="3.404194228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:35:59.329848745 +0000 UTC m=+336.996035167" watchObservedRunningTime="2026-07-14 14:35:59.33108547 +0000 UTC m=+336.997271892"
Jul 14 14:37:56.142205 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.142116    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"]
Jul 14 14:37:56.145530 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.145507    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"
Jul 14 14:37:56.148313 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.148291    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 14 14:37:56.148431 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.148313    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fzsbb\""
Jul 14 14:37:56.148431 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.148295    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 14 14:37:56.160356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.160327    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"]
Jul 14 14:37:56.274786 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.274752    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/214dec35-0b45-4763-8a01-cb485ace1243-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lhh2n\" (UID: \"214dec35-0b45-4763-8a01-cb485ace1243\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"
Jul 14 14:37:56.274958 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.274836    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7vhl\" (UniqueName: \"kubernetes.io/projected/214dec35-0b45-4763-8a01-cb485ace1243-kube-api-access-j7vhl\") pod \"kuadrant-operator-controller-manager-86878c769f-lhh2n\" (UID: \"214dec35-0b45-4763-8a01-cb485ace1243\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"
Jul 14 14:37:56.376207 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.376169    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/214dec35-0b45-4763-8a01-cb485ace1243-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lhh2n\" (UID: \"214dec35-0b45-4763-8a01-cb485ace1243\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"
Jul 14 14:37:56.376385 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.376231    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7vhl\" (UniqueName: \"kubernetes.io/projected/214dec35-0b45-4763-8a01-cb485ace1243-kube-api-access-j7vhl\") pod \"kuadrant-operator-controller-manager-86878c769f-lhh2n\" (UID: \"214dec35-0b45-4763-8a01-cb485ace1243\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"
Jul 14 14:37:56.376610 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.376588    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/214dec35-0b45-4763-8a01-cb485ace1243-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lhh2n\" (UID: \"214dec35-0b45-4763-8a01-cb485ace1243\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"
Jul 14 14:37:56.387070 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.387039    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7vhl\" (UniqueName: \"kubernetes.io/projected/214dec35-0b45-4763-8a01-cb485ace1243-kube-api-access-j7vhl\") pod \"kuadrant-operator-controller-manager-86878c769f-lhh2n\" (UID: \"214dec35-0b45-4763-8a01-cb485ace1243\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"
Jul 14 14:37:56.455777 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.455748    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"
Jul 14 14:37:56.593524 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.593496    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"]
Jul 14 14:37:56.596290 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:37:56.596263    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214dec35_0b45_4763_8a01_cb485ace1243.slice/crio-5a77bdbf4618cfd5cdfedb19eb69364d3fb1016cca6b4fb99f0de55a879e4d58 WatchSource:0}: Error finding container 5a77bdbf4618cfd5cdfedb19eb69364d3fb1016cca6b4fb99f0de55a879e4d58: Status 404 returned error can't find the container with id 5a77bdbf4618cfd5cdfedb19eb69364d3fb1016cca6b4fb99f0de55a879e4d58
Jul 14 14:37:56.823831 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:37:56.823746    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n" event={"ID":"214dec35-0b45-4763-8a01-cb485ace1243","Type":"ContainerStarted","Data":"5a77bdbf4618cfd5cdfedb19eb69364d3fb1016cca6b4fb99f0de55a879e4d58"}
Jul 14 14:38:01.854048 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:01.854006    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n" event={"ID":"214dec35-0b45-4763-8a01-cb485ace1243","Type":"ContainerStarted","Data":"b70525b48481b2bd00de607f9be1f1c0c2a84f96b22565edfd9366aff762ddcf"}
Jul 14 14:38:01.854513 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:01.854178    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"
Jul 14 14:38:01.876509 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:01.876450    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n" podStartSLOduration=0.842314587 podStartE2EDuration="5.876432438s" totalImagesPullingTime="5.034117851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:38:01.874995748 +0000 UTC m=+459.541182168" watchObservedRunningTime="2026-07-14 14:38:01.876432438 +0000 UTC m=+459.542618856"
Jul 14 14:38:12.861715 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:12.861681    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"
Jul 14 14:38:32.356851 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:32.356813    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-nzht5"]
Jul 14 14:38:32.364281 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:32.364255    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nzht5"
Jul 14 14:38:32.367170 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:32.367148    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9gjbq\""
Jul 14 14:38:32.371167 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:32.371142    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nzht5"]
Jul 14 14:38:32.391596 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:32.391562    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc6ln\" (UniqueName: \"kubernetes.io/projected/59cb0fb4-ed68-42bb-88da-aa807891e12f-kube-api-access-bc6ln\") pod \"authorino-75d556bd77-nzht5\" (UID: \"59cb0fb4-ed68-42bb-88da-aa807891e12f\") " pod="kuadrant-system/authorino-75d556bd77-nzht5"
Jul 14 14:38:32.492068 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:32.492031    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc6ln\" (UniqueName: \"kubernetes.io/projected/59cb0fb4-ed68-42bb-88da-aa807891e12f-kube-api-access-bc6ln\") pod \"authorino-75d556bd77-nzht5\" (UID: \"59cb0fb4-ed68-42bb-88da-aa807891e12f\") " pod="kuadrant-system/authorino-75d556bd77-nzht5"
Jul 14 14:38:32.500773 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:32.500747    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc6ln\" (UniqueName: \"kubernetes.io/projected/59cb0fb4-ed68-42bb-88da-aa807891e12f-kube-api-access-bc6ln\") pod \"authorino-75d556bd77-nzht5\" (UID: \"59cb0fb4-ed68-42bb-88da-aa807891e12f\") " pod="kuadrant-system/authorino-75d556bd77-nzht5"
Jul 14 14:38:32.686155 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:32.686121    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nzht5"
Jul 14 14:38:32.840643 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:32.840606    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nzht5"]
Jul 14 14:38:32.845497 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:38:32.845465    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59cb0fb4_ed68_42bb_88da_aa807891e12f.slice/crio-259fcc9bb818b9144de1a717b961bf65a2803bcd9cab9916f870d754a6aa3572 WatchSource:0}: Error finding container 259fcc9bb818b9144de1a717b961bf65a2803bcd9cab9916f870d754a6aa3572: Status 404 returned error can't find the container with id 259fcc9bb818b9144de1a717b961bf65a2803bcd9cab9916f870d754a6aa3572
Jul 14 14:38:32.992880 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:32.992795    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nzht5" event={"ID":"59cb0fb4-ed68-42bb-88da-aa807891e12f","Type":"ContainerStarted","Data":"259fcc9bb818b9144de1a717b961bf65a2803bcd9cab9916f870d754a6aa3572"}
Jul 14 14:38:37.026024 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:37.025989    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nzht5" event={"ID":"59cb0fb4-ed68-42bb-88da-aa807891e12f","Type":"ContainerStarted","Data":"99fbb6a7c2777551d263c5fe692cf540d7b48cd26670555b5329d78969cb8ff5"}
Jul 14 14:38:37.042530 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:38:37.042479    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-nzht5" podStartSLOduration=1.561133238 podStartE2EDuration="5.042464105s" totalImagesPullingTime="3.481330867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:38:37.041735065 +0000 UTC m=+494.707921484" watchObservedRunningTime="2026-07-14 14:38:37.042464105 +0000 UTC m=+494.708650530"
Jul 14 14:39:02.451246 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:02.451204    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nzht5"]
Jul 14 14:39:02.451729 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:02.451464    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-nzht5" podUID="59cb0fb4-ed68-42bb-88da-aa807891e12f" containerName="authorino" containerID="cri-o://99fbb6a7c2777551d263c5fe692cf540d7b48cd26670555b5329d78969cb8ff5" gracePeriod=30
Jul 14 14:39:02.703380 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:02.703324    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nzht5"
Jul 14 14:39:02.859243 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:02.859207    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc6ln\" (UniqueName: \"kubernetes.io/projected/59cb0fb4-ed68-42bb-88da-aa807891e12f-kube-api-access-bc6ln\") pod \"59cb0fb4-ed68-42bb-88da-aa807891e12f\" (UID: \"59cb0fb4-ed68-42bb-88da-aa807891e12f\") "
Jul 14 14:39:02.861541 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:02.861516    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59cb0fb4-ed68-42bb-88da-aa807891e12f-kube-api-access-bc6ln" (OuterVolumeSpecName: "kube-api-access-bc6ln") pod "59cb0fb4-ed68-42bb-88da-aa807891e12f" (UID: "59cb0fb4-ed68-42bb-88da-aa807891e12f"). InnerVolumeSpecName "kube-api-access-bc6ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:39:02.960946 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:02.960865    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc6ln\" (UniqueName: \"kubernetes.io/projected/59cb0fb4-ed68-42bb-88da-aa807891e12f-kube-api-access-bc6ln\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:39:03.143507 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:03.143468    2586 generic.go:356] "Generic (PLEG): container finished" podID="59cb0fb4-ed68-42bb-88da-aa807891e12f" containerID="99fbb6a7c2777551d263c5fe692cf540d7b48cd26670555b5329d78969cb8ff5" exitCode=0
Jul 14 14:39:03.143507 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:03.143508    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nzht5"
Jul 14 14:39:03.143851 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:03.143507    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nzht5" event={"ID":"59cb0fb4-ed68-42bb-88da-aa807891e12f","Type":"ContainerDied","Data":"99fbb6a7c2777551d263c5fe692cf540d7b48cd26670555b5329d78969cb8ff5"}
Jul 14 14:39:03.143851 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:03.143560    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nzht5" event={"ID":"59cb0fb4-ed68-42bb-88da-aa807891e12f","Type":"ContainerDied","Data":"259fcc9bb818b9144de1a717b961bf65a2803bcd9cab9916f870d754a6aa3572"}
Jul 14 14:39:03.143851 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:03.143581    2586 scope.go:117] "RemoveContainer" containerID="99fbb6a7c2777551d263c5fe692cf540d7b48cd26670555b5329d78969cb8ff5"
Jul 14 14:39:03.151866 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:03.151848    2586 scope.go:117] "RemoveContainer" containerID="99fbb6a7c2777551d263c5fe692cf540d7b48cd26670555b5329d78969cb8ff5"
Jul 14 14:39:03.152130 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:39:03.152111    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99fbb6a7c2777551d263c5fe692cf540d7b48cd26670555b5329d78969cb8ff5\": container with ID starting with 99fbb6a7c2777551d263c5fe692cf540d7b48cd26670555b5329d78969cb8ff5 not found: ID does not exist" containerID="99fbb6a7c2777551d263c5fe692cf540d7b48cd26670555b5329d78969cb8ff5"
Jul 14 14:39:03.152176 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:03.152143    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fbb6a7c2777551d263c5fe692cf540d7b48cd26670555b5329d78969cb8ff5"} err="failed to get container status \"99fbb6a7c2777551d263c5fe692cf540d7b48cd26670555b5329d78969cb8ff5\": rpc error: code = NotFound desc = could not find container \"99fbb6a7c2777551d263c5fe692cf540d7b48cd26670555b5329d78969cb8ff5\": container with ID starting with 99fbb6a7c2777551d263c5fe692cf540d7b48cd26670555b5329d78969cb8ff5 not found: ID does not exist"
Jul 14 14:39:03.167205 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:03.167183    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nzht5"]
Jul 14 14:39:03.175500 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:03.175478    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nzht5"]
Jul 14 14:39:04.875653 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:04.875620    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59cb0fb4-ed68-42bb-88da-aa807891e12f" path="/var/lib/kubelet/pods/59cb0fb4-ed68-42bb-88da-aa807891e12f/volumes"
Jul 14 14:39:07.141540 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.141511    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b449c959f-7t9jh"]
Jul 14 14:39:07.141879 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.141851    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59cb0fb4-ed68-42bb-88da-aa807891e12f" containerName="authorino"
Jul 14 14:39:07.141879 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.141863    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="59cb0fb4-ed68-42bb-88da-aa807891e12f" containerName="authorino"
Jul 14 14:39:07.141949 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.141934    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="59cb0fb4-ed68-42bb-88da-aa807891e12f" containerName="authorino"
Jul 14 14:39:07.146133 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.146113    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b449c959f-7t9jh"
Jul 14 14:39:07.149106 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.149044    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 14 14:39:07.149232 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.149207    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sn8zt\""
Jul 14 14:39:07.149232 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.149217    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 14 14:39:07.153600 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.153547    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b449c959f-7t9jh"]
Jul 14 14:39:07.197910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.197882    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/098dacfa-8fbe-447a-a7ec-7873037da51b-webhook-cert\") pod \"maas-controller-b449c959f-7t9jh\" (UID: \"098dacfa-8fbe-447a-a7ec-7873037da51b\") " pod="opendatahub/maas-controller-b449c959f-7t9jh"
Jul 14 14:39:07.198081 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.197930    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vs5k\" (UniqueName: \"kubernetes.io/projected/098dacfa-8fbe-447a-a7ec-7873037da51b-kube-api-access-6vs5k\") pod \"maas-controller-b449c959f-7t9jh\" (UID: \"098dacfa-8fbe-447a-a7ec-7873037da51b\") " pod="opendatahub/maas-controller-b449c959f-7t9jh"
Jul 14 14:39:07.299321 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.299289    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/098dacfa-8fbe-447a-a7ec-7873037da51b-webhook-cert\") pod \"maas-controller-b449c959f-7t9jh\" (UID: \"098dacfa-8fbe-447a-a7ec-7873037da51b\") " pod="opendatahub/maas-controller-b449c959f-7t9jh"
Jul 14 14:39:07.299544 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.299348    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vs5k\" (UniqueName: \"kubernetes.io/projected/098dacfa-8fbe-447a-a7ec-7873037da51b-kube-api-access-6vs5k\") pod \"maas-controller-b449c959f-7t9jh\" (UID: \"098dacfa-8fbe-447a-a7ec-7873037da51b\") " pod="opendatahub/maas-controller-b449c959f-7t9jh"
Jul 14 14:39:07.299544 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:39:07.299498    2586 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 14 14:39:07.299673 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:39:07.299576    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098dacfa-8fbe-447a-a7ec-7873037da51b-webhook-cert podName:098dacfa-8fbe-447a-a7ec-7873037da51b nodeName:}" failed. No retries permitted until 2026-07-14 14:39:07.799556329 +0000 UTC m=+525.465742732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/098dacfa-8fbe-447a-a7ec-7873037da51b-webhook-cert") pod "maas-controller-b449c959f-7t9jh" (UID: "098dacfa-8fbe-447a-a7ec-7873037da51b") : secret "maas-controller-webhook-cert" not found
Jul 14 14:39:07.312359 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.312334    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vs5k\" (UniqueName: \"kubernetes.io/projected/098dacfa-8fbe-447a-a7ec-7873037da51b-kube-api-access-6vs5k\") pod \"maas-controller-b449c959f-7t9jh\" (UID: \"098dacfa-8fbe-447a-a7ec-7873037da51b\") " pod="opendatahub/maas-controller-b449c959f-7t9jh"
Jul 14 14:39:07.320805 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.320781    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5898bfff6b-z28q4"]
Jul 14 14:39:07.324114 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.324095    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5898bfff6b-z28q4"
Jul 14 14:39:07.332234 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.332213    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5898bfff6b-z28q4"]
Jul 14 14:39:07.400589 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.400500    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpd6x\" (UniqueName: \"kubernetes.io/projected/348b64ef-bcd5-4eb5-98e4-f30f812d7251-kube-api-access-fpd6x\") pod \"maas-controller-5898bfff6b-z28q4\" (UID: \"348b64ef-bcd5-4eb5-98e4-f30f812d7251\") " pod="opendatahub/maas-controller-5898bfff6b-z28q4"
Jul 14 14:39:07.400745 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.400592    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/348b64ef-bcd5-4eb5-98e4-f30f812d7251-webhook-cert\") pod \"maas-controller-5898bfff6b-z28q4\" (UID: \"348b64ef-bcd5-4eb5-98e4-f30f812d7251\") " pod="opendatahub/maas-controller-5898bfff6b-z28q4"
Jul 14 14:39:07.501345 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.501315    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/348b64ef-bcd5-4eb5-98e4-f30f812d7251-webhook-cert\") pod \"maas-controller-5898bfff6b-z28q4\" (UID: \"348b64ef-bcd5-4eb5-98e4-f30f812d7251\") " pod="opendatahub/maas-controller-5898bfff6b-z28q4"
Jul 14 14:39:07.501559 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.501436    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpd6x\" (UniqueName: \"kubernetes.io/projected/348b64ef-bcd5-4eb5-98e4-f30f812d7251-kube-api-access-fpd6x\") pod \"maas-controller-5898bfff6b-z28q4\" (UID: \"348b64ef-bcd5-4eb5-98e4-f30f812d7251\") " pod="opendatahub/maas-controller-5898bfff6b-z28q4"
Jul 14 14:39:07.503924 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.503906    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/348b64ef-bcd5-4eb5-98e4-f30f812d7251-webhook-cert\") pod \"maas-controller-5898bfff6b-z28q4\" (UID: \"348b64ef-bcd5-4eb5-98e4-f30f812d7251\") " pod="opendatahub/maas-controller-5898bfff6b-z28q4"
Jul 14 14:39:07.510298 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.510271    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpd6x\" (UniqueName: \"kubernetes.io/projected/348b64ef-bcd5-4eb5-98e4-f30f812d7251-kube-api-access-fpd6x\") pod \"maas-controller-5898bfff6b-z28q4\" (UID: \"348b64ef-bcd5-4eb5-98e4-f30f812d7251\") " pod="opendatahub/maas-controller-5898bfff6b-z28q4"
Jul 14 14:39:07.635115 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.635080    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5898bfff6b-z28q4"
Jul 14 14:39:07.795480 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.795452    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5898bfff6b-z28q4"]
Jul 14 14:39:07.798106 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:39:07.798078    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348b64ef_bcd5_4eb5_98e4_f30f812d7251.slice/crio-a8f3462e2b95c11059d0e486d6d8d9fc74a61b3a0aa2bccc2c4901cc4c3ad7e0 WatchSource:0}: Error finding container a8f3462e2b95c11059d0e486d6d8d9fc74a61b3a0aa2bccc2c4901cc4c3ad7e0: Status 404 returned error can't find the container with id a8f3462e2b95c11059d0e486d6d8d9fc74a61b3a0aa2bccc2c4901cc4c3ad7e0
Jul 14 14:39:07.805155 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.805133    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/098dacfa-8fbe-447a-a7ec-7873037da51b-webhook-cert\") pod \"maas-controller-b449c959f-7t9jh\" (UID: \"098dacfa-8fbe-447a-a7ec-7873037da51b\") " pod="opendatahub/maas-controller-b449c959f-7t9jh"
Jul 14 14:39:07.807497 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:07.807480    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/098dacfa-8fbe-447a-a7ec-7873037da51b-webhook-cert\") pod \"maas-controller-b449c959f-7t9jh\" (UID: \"098dacfa-8fbe-447a-a7ec-7873037da51b\") " pod="opendatahub/maas-controller-b449c959f-7t9jh"
Jul 14 14:39:08.057449 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:08.057359    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b449c959f-7t9jh"
Jul 14 14:39:08.178814 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:08.178780    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5898bfff6b-z28q4" event={"ID":"348b64ef-bcd5-4eb5-98e4-f30f812d7251","Type":"ContainerStarted","Data":"a8f3462e2b95c11059d0e486d6d8d9fc74a61b3a0aa2bccc2c4901cc4c3ad7e0"}
Jul 14 14:39:08.193699 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:08.193677    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b449c959f-7t9jh"]
Jul 14 14:39:08.195882 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:39:08.195851    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098dacfa_8fbe_447a_a7ec_7873037da51b.slice/crio-57d8232265d08ae330f91da378c1999f77109aa7be20cc5c9bad773e5ab8da6a WatchSource:0}: Error finding container 57d8232265d08ae330f91da378c1999f77109aa7be20cc5c9bad773e5ab8da6a: Status 404 returned error can't find the container with id 57d8232265d08ae330f91da378c1999f77109aa7be20cc5c9bad773e5ab8da6a
Jul 14 14:39:09.192176 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:09.189869    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b449c959f-7t9jh" event={"ID":"098dacfa-8fbe-447a-a7ec-7873037da51b","Type":"ContainerStarted","Data":"57d8232265d08ae330f91da378c1999f77109aa7be20cc5c9bad773e5ab8da6a"}
Jul 14 14:39:11.202796 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:11.202687    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5898bfff6b-z28q4" event={"ID":"348b64ef-bcd5-4eb5-98e4-f30f812d7251","Type":"ContainerStarted","Data":"f1f6ab896d7acc699806c7d2b7cfd4b6a107cb499f1bafb8da067723b09c2ce1"}
Jul 14 14:39:11.202796 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:11.202751    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5898bfff6b-z28q4"
Jul 14 14:39:11.204214 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:11.204191    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b449c959f-7t9jh" event={"ID":"098dacfa-8fbe-447a-a7ec-7873037da51b","Type":"ContainerStarted","Data":"08888fe01900ecccc2bd95e59e95cb93033f813c5820b94949a4fc5197e9a8ce"}
Jul 14 14:39:11.204340 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:11.204328    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b449c959f-7t9jh"
Jul 14 14:39:11.224543 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:11.224495    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5898bfff6b-z28q4" podStartSLOduration=1.07830181 podStartE2EDuration="4.224479921s" totalImagesPullingTime="3.146178111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:39:11.223246555 +0000 UTC m=+528.889432977" watchObservedRunningTime="2026-07-14 14:39:11.224479921 +0000 UTC m=+528.890666339"
Jul 14 14:39:11.242136 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:11.242096    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b449c959f-7t9jh" podStartSLOduration=1.489304722 podStartE2EDuration="4.242057288s" totalImagesPullingTime="2.752752566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:39:11.241168529 +0000 UTC m=+528.907354949" watchObservedRunningTime="2026-07-14 14:39:11.242057288 +0000 UTC m=+528.908243708"
Jul 14 14:39:22.210819 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:22.210747    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b449c959f-7t9jh"
Jul 14 14:39:22.211199 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:22.210925    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5898bfff6b-z28q4"
Jul 14 14:39:22.274447 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:22.274406    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b449c959f-7t9jh"]
Jul 14 14:39:22.274668 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:22.274645    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-b449c959f-7t9jh" podUID="098dacfa-8fbe-447a-a7ec-7873037da51b" containerName="manager" containerID="cri-o://08888fe01900ecccc2bd95e59e95cb93033f813c5820b94949a4fc5197e9a8ce" gracePeriod=30
Jul 14 14:39:22.516902 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:22.516872    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b449c959f-7t9jh"
Jul 14 14:39:22.636553 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:22.636514    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/098dacfa-8fbe-447a-a7ec-7873037da51b-webhook-cert\") pod \"098dacfa-8fbe-447a-a7ec-7873037da51b\" (UID: \"098dacfa-8fbe-447a-a7ec-7873037da51b\") "
Jul 14 14:39:22.636747 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:22.636622    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vs5k\" (UniqueName: \"kubernetes.io/projected/098dacfa-8fbe-447a-a7ec-7873037da51b-kube-api-access-6vs5k\") pod \"098dacfa-8fbe-447a-a7ec-7873037da51b\" (UID: \"098dacfa-8fbe-447a-a7ec-7873037da51b\") "
Jul 14 14:39:22.638817 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:22.638779    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/098dacfa-8fbe-447a-a7ec-7873037da51b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "098dacfa-8fbe-447a-a7ec-7873037da51b" (UID: "098dacfa-8fbe-447a-a7ec-7873037da51b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:39:22.638957 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:22.638835    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098dacfa-8fbe-447a-a7ec-7873037da51b-kube-api-access-6vs5k" (OuterVolumeSpecName: "kube-api-access-6vs5k") pod "098dacfa-8fbe-447a-a7ec-7873037da51b" (UID: "098dacfa-8fbe-447a-a7ec-7873037da51b"). InnerVolumeSpecName "kube-api-access-6vs5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:39:22.737320 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:22.737290    2586 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/098dacfa-8fbe-447a-a7ec-7873037da51b-webhook-cert\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:39:22.737320 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:22.737314    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vs5k\" (UniqueName: \"kubernetes.io/projected/098dacfa-8fbe-447a-a7ec-7873037da51b-kube-api-access-6vs5k\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:39:23.260073 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:23.260014    2586 generic.go:356] "Generic (PLEG): container finished" podID="098dacfa-8fbe-447a-a7ec-7873037da51b" containerID="08888fe01900ecccc2bd95e59e95cb93033f813c5820b94949a4fc5197e9a8ce" exitCode=0
Jul 14 14:39:23.260461 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:23.260086    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b449c959f-7t9jh"
Jul 14 14:39:23.260461 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:23.260096    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b449c959f-7t9jh" event={"ID":"098dacfa-8fbe-447a-a7ec-7873037da51b","Type":"ContainerDied","Data":"08888fe01900ecccc2bd95e59e95cb93033f813c5820b94949a4fc5197e9a8ce"}
Jul 14 14:39:23.260461 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:23.260174    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b449c959f-7t9jh" event={"ID":"098dacfa-8fbe-447a-a7ec-7873037da51b","Type":"ContainerDied","Data":"57d8232265d08ae330f91da378c1999f77109aa7be20cc5c9bad773e5ab8da6a"}
Jul 14 14:39:23.260461 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:23.260196    2586 scope.go:117] "RemoveContainer" containerID="08888fe01900ecccc2bd95e59e95cb93033f813c5820b94949a4fc5197e9a8ce"
Jul 14 14:39:23.270345 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:23.270310    2586 scope.go:117] "RemoveContainer" containerID="08888fe01900ecccc2bd95e59e95cb93033f813c5820b94949a4fc5197e9a8ce"
Jul 14 14:39:23.270698 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:39:23.270677    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08888fe01900ecccc2bd95e59e95cb93033f813c5820b94949a4fc5197e9a8ce\": container with ID starting with 08888fe01900ecccc2bd95e59e95cb93033f813c5820b94949a4fc5197e9a8ce not found: ID does not exist" containerID="08888fe01900ecccc2bd95e59e95cb93033f813c5820b94949a4fc5197e9a8ce"
Jul 14 14:39:23.270785 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:23.270710    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08888fe01900ecccc2bd95e59e95cb93033f813c5820b94949a4fc5197e9a8ce"} err="failed to get container status \"08888fe01900ecccc2bd95e59e95cb93033f813c5820b94949a4fc5197e9a8ce\": rpc error: code = NotFound desc = could not find container \"08888fe01900ecccc2bd95e59e95cb93033f813c5820b94949a4fc5197e9a8ce\": container with ID starting with 08888fe01900ecccc2bd95e59e95cb93033f813c5820b94949a4fc5197e9a8ce not found: ID does not exist"
Jul 14 14:39:23.282483 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:23.282461    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b449c959f-7t9jh"]
Jul 14 14:39:23.288710 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:23.288690    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b449c959f-7t9jh"]
Jul 14 14:39:24.225462 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.225405    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"]
Jul 14 14:39:24.225889 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.225871    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="098dacfa-8fbe-447a-a7ec-7873037da51b" containerName="manager"
Jul 14 14:39:24.225973 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.225891    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="098dacfa-8fbe-447a-a7ec-7873037da51b" containerName="manager"
Jul 14 14:39:24.226021 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.225989    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="098dacfa-8fbe-447a-a7ec-7873037da51b" containerName="manager"
Jul 14 14:39:24.230213 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.230192    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"
Jul 14 14:39:24.233521 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.233498    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kzncc\""
Jul 14 14:39:24.234648 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.234629    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 14 14:39:24.234745 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.234673    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 14:39:24.234745 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.234677    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 14:39:24.243202 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.243179    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"]
Jul 14 14:39:24.350807 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.350777    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3-openshift-service-ca\") pod \"maas-api-5b4d9fd998-qx6v9\" (UID: \"aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3\") " pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"
Jul 14 14:39:24.351318 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.350816    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c486w\" (UniqueName: \"kubernetes.io/projected/aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3-kube-api-access-c486w\") pod \"maas-api-5b4d9fd998-qx6v9\" (UID: \"aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3\") " pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"
Jul 14 14:39:24.351318 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.350900    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3-maas-api-tls\") pod \"maas-api-5b4d9fd998-qx6v9\" (UID: \"aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3\") " pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"
Jul 14 14:39:24.452343 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.452308    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3-openshift-service-ca\") pod \"maas-api-5b4d9fd998-qx6v9\" (UID: \"aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3\") " pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"
Jul 14 14:39:24.452343 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.452344    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c486w\" (UniqueName: \"kubernetes.io/projected/aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3-kube-api-access-c486w\") pod \"maas-api-5b4d9fd998-qx6v9\" (UID: \"aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3\") " pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"
Jul 14 14:39:24.452614 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.452382    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3-maas-api-tls\") pod \"maas-api-5b4d9fd998-qx6v9\" (UID: \"aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3\") " pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"
Jul 14 14:39:24.453082 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.453058    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3-openshift-service-ca\") pod \"maas-api-5b4d9fd998-qx6v9\" (UID: \"aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3\") " pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"
Jul 14 14:39:24.454813 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.454795    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3-maas-api-tls\") pod \"maas-api-5b4d9fd998-qx6v9\" (UID: \"aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3\") " pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"
Jul 14 14:39:24.466795 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.466757    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c486w\" (UniqueName: \"kubernetes.io/projected/aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3-kube-api-access-c486w\") pod \"maas-api-5b4d9fd998-qx6v9\" (UID: \"aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3\") " pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"
Jul 14 14:39:24.540899 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.540805    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"
Jul 14 14:39:24.679838 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.679810    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"]
Jul 14 14:39:24.682470 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:39:24.682441    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1e3107_9226_4b8d_8d2c_a30a94c1a6e3.slice/crio-6296a618113d1d7090e5f1d2294a142a0e3f02a86979874d120f292129e66820 WatchSource:0}: Error finding container 6296a618113d1d7090e5f1d2294a142a0e3f02a86979874d120f292129e66820: Status 404 returned error can't find the container with id 6296a618113d1d7090e5f1d2294a142a0e3f02a86979874d120f292129e66820
Jul 14 14:39:24.876612 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:24.876541    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="098dacfa-8fbe-447a-a7ec-7873037da51b" path="/var/lib/kubelet/pods/098dacfa-8fbe-447a-a7ec-7873037da51b/volumes"
Jul 14 14:39:25.271502 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:25.271466    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9" event={"ID":"aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3","Type":"ContainerStarted","Data":"6296a618113d1d7090e5f1d2294a142a0e3f02a86979874d120f292129e66820"}
Jul 14 14:39:27.282805 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:27.282765    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9" event={"ID":"aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3","Type":"ContainerStarted","Data":"2b7c1e64945e0fc085569f8e8468bef13541be8d84aaefa19b2cca31b74a69c2"}
Jul 14 14:39:27.283254 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:27.282902    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"
Jul 14 14:39:27.303545 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:27.303498    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9" podStartSLOduration=1.569579612 podStartE2EDuration="3.303482078s" totalImagesPullingTime="1.733902466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:39:27.3014577 +0000 UTC m=+544.967644125" watchObservedRunningTime="2026-07-14 14:39:27.303482078 +0000 UTC m=+544.969668499"
Jul 14 14:39:33.299199 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:33.299168    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5b4d9fd998-qx6v9"
Jul 14 14:39:44.068271 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.068233    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"]
Jul 14 14:39:44.072845 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.072827    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.076723 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.076698    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tqhk5\""
Jul 14 14:39:44.076927 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.076911    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 14 14:39:44.077080 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.077068    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 14 14:39:44.077295 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.077275    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 14 14:39:44.079998 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.079966    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"]
Jul 14 14:39:44.231400 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.231364    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d661f6cc-1b4e-4ba5-980c-911e406264bd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.231581 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.231446    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d661f6cc-1b4e-4ba5-980c-911e406264bd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.231581 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.231488    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vk58\" (UniqueName: \"kubernetes.io/projected/d661f6cc-1b4e-4ba5-980c-911e406264bd-kube-api-access-8vk58\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.231581 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.231567    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d661f6cc-1b4e-4ba5-980c-911e406264bd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.231704 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.231595    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d661f6cc-1b4e-4ba5-980c-911e406264bd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.231704 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.231628    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d661f6cc-1b4e-4ba5-980c-911e406264bd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.332442 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.332328    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d661f6cc-1b4e-4ba5-980c-911e406264bd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.332442 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.332390    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d661f6cc-1b4e-4ba5-980c-911e406264bd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.332731 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.332463    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vk58\" (UniqueName: \"kubernetes.io/projected/d661f6cc-1b4e-4ba5-980c-911e406264bd-kube-api-access-8vk58\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.332731 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.332514    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d661f6cc-1b4e-4ba5-980c-911e406264bd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.332731 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.332537    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d661f6cc-1b4e-4ba5-980c-911e406264bd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.332731 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.332574    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d661f6cc-1b4e-4ba5-980c-911e406264bd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.332943 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.332829    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d661f6cc-1b4e-4ba5-980c-911e406264bd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.332943 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.332840    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d661f6cc-1b4e-4ba5-980c-911e406264bd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.333034 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.332970    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d661f6cc-1b4e-4ba5-980c-911e406264bd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.334725 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.334703    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d661f6cc-1b4e-4ba5-980c-911e406264bd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.335083 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.335064    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d661f6cc-1b4e-4ba5-980c-911e406264bd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.340888 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.340868    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vk58\" (UniqueName: \"kubernetes.io/projected/d661f6cc-1b4e-4ba5-980c-911e406264bd-kube-api-access-8vk58\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx\" (UID: \"d661f6cc-1b4e-4ba5-980c-911e406264bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.383236 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.383201    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:44.520191 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:44.520155    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"]
Jul 14 14:39:44.522481 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:39:44.522454    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd661f6cc_1b4e_4ba5_980c_911e406264bd.slice/crio-21662f0adda4363c1b691b14d150c613989e96b047456a55218b715398bb8b72 WatchSource:0}: Error finding container 21662f0adda4363c1b691b14d150c613989e96b047456a55218b715398bb8b72: Status 404 returned error can't find the container with id 21662f0adda4363c1b691b14d150c613989e96b047456a55218b715398bb8b72
Jul 14 14:39:45.371181 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:45.371143    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx" event={"ID":"d661f6cc-1b4e-4ba5-980c-911e406264bd","Type":"ContainerStarted","Data":"21662f0adda4363c1b691b14d150c613989e96b047456a55218b715398bb8b72"}
Jul 14 14:39:46.377557 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:46.377519    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx" event={"ID":"d661f6cc-1b4e-4ba5-980c-911e406264bd","Type":"ContainerStarted","Data":"97c42634a48ab9691b5b864c5c55b1738015acf20b7cc1452b27031409538abe"}
Jul 14 14:39:46.395293 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:46.395238    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx" podStartSLOduration=1.291116 podStartE2EDuration="2.395217925s" totalImagesPullingTime="1.104101925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:39:46.394521043 +0000 UTC m=+564.060707465" watchObservedRunningTime="2026-07-14 14:39:46.395217925 +0000 UTC m=+564.061404347"
Jul 14 14:39:54.384238 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:54.384194    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:54.384667 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:54.384273    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:54.397238 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:54.397215    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:39:54.427944 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:39:54.427916    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx"
Jul 14 14:40:00.752870 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.752837    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"]
Jul 14 14:40:00.757356 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.757334    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.760357 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.760338    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 14 14:40:00.768191 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.768165    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"]
Jul 14 14:40:00.871945 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.871911    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5117094c-0aa4-4043-a8a7-88cf03e92a64-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.872119 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.872010    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5117094c-0aa4-4043-a8a7-88cf03e92a64-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.872119 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.872042    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5117094c-0aa4-4043-a8a7-88cf03e92a64-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.872119 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.872057    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5117094c-0aa4-4043-a8a7-88cf03e92a64-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.872119 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.872086    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c25ck\" (UniqueName: \"kubernetes.io/projected/5117094c-0aa4-4043-a8a7-88cf03e92a64-kube-api-access-c25ck\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.872119 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.872118    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5117094c-0aa4-4043-a8a7-88cf03e92a64-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.973210 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.973178    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5117094c-0aa4-4043-a8a7-88cf03e92a64-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.973396 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.973242    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5117094c-0aa4-4043-a8a7-88cf03e92a64-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.973396 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.973268    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5117094c-0aa4-4043-a8a7-88cf03e92a64-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.973396 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.973290    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5117094c-0aa4-4043-a8a7-88cf03e92a64-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.973396 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.973313    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c25ck\" (UniqueName: \"kubernetes.io/projected/5117094c-0aa4-4043-a8a7-88cf03e92a64-kube-api-access-c25ck\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.973396 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.973353    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5117094c-0aa4-4043-a8a7-88cf03e92a64-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.973689 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.973669    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5117094c-0aa4-4043-a8a7-88cf03e92a64-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.973737 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.973686    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5117094c-0aa4-4043-a8a7-88cf03e92a64-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.973832 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.973811    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5117094c-0aa4-4043-a8a7-88cf03e92a64-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.975811 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.975774    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5117094c-0aa4-4043-a8a7-88cf03e92a64-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.976003 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.975987    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5117094c-0aa4-4043-a8a7-88cf03e92a64-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:00.982291 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:00.982270    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c25ck\" (UniqueName: \"kubernetes.io/projected/5117094c-0aa4-4043-a8a7-88cf03e92a64-kube-api-access-c25ck\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d4fk6\" (UID: \"5117094c-0aa4-4043-a8a7-88cf03e92a64\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:01.068480 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:01.068367    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:01.205324 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:01.205295    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"]
Jul 14 14:40:01.209296 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:40:01.209257    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5117094c_0aa4_4043_a8a7_88cf03e92a64.slice/crio-67bd60be81171b69fa2b27b390d39cd88a99a6cc9bb817cf8ec701807fa0487a WatchSource:0}: Error finding container 67bd60be81171b69fa2b27b390d39cd88a99a6cc9bb817cf8ec701807fa0487a: Status 404 returned error can't find the container with id 67bd60be81171b69fa2b27b390d39cd88a99a6cc9bb817cf8ec701807fa0487a
Jul 14 14:40:01.454664 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:01.454630    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6" event={"ID":"5117094c-0aa4-4043-a8a7-88cf03e92a64","Type":"ContainerStarted","Data":"67bd60be81171b69fa2b27b390d39cd88a99a6cc9bb817cf8ec701807fa0487a"}
Jul 14 14:40:02.460862 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:02.460827    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6" event={"ID":"5117094c-0aa4-4043-a8a7-88cf03e92a64","Type":"ContainerStarted","Data":"14596e5de619236508a93bef7962b86350b552704f67c1c0bb21cec6a11f2dd0"}
Jul 14 14:40:02.483030 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:02.482971    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6" podStartSLOduration=2.28721122 podStartE2EDuration="2.482955026s" totalImagesPullingTime="195.743806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:40:02.481137119 +0000 UTC m=+580.147323542" watchObservedRunningTime="2026-07-14 14:40:02.482955026 +0000 UTC m=+580.149141446"
Jul 14 14:40:11.069124 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:11.069080    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:11.069543 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:11.069141    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:11.081700 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:11.081679    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:11.515273 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:11.515241    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d4fk6"
Jul 14 14:40:22.801866 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:22.801839    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:40:22.809529 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:40:22.809511    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:41:14.833274 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:14.833238    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b68b66c55-74sdf"]
Jul 14 14:41:14.836782 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:14.836761    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b68b66c55-74sdf"
Jul 14 14:41:14.840839 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:14.840815    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 14 14:41:14.840981 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:14.840815    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9gjbq\""
Jul 14 14:41:14.844154 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:14.843990    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b68b66c55-74sdf"]
Jul 14 14:41:14.936424 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:14.936388    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d3edc5b-855e-46ba-9585-55ee5665fa97-tls-cert\") pod \"authorino-7b68b66c55-74sdf\" (UID: \"5d3edc5b-855e-46ba-9585-55ee5665fa97\") " pod="kuadrant-system/authorino-7b68b66c55-74sdf"
Jul 14 14:41:14.936586 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:14.936501    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmg8j\" (UniqueName: \"kubernetes.io/projected/5d3edc5b-855e-46ba-9585-55ee5665fa97-kube-api-access-gmg8j\") pod \"authorino-7b68b66c55-74sdf\" (UID: \"5d3edc5b-855e-46ba-9585-55ee5665fa97\") " pod="kuadrant-system/authorino-7b68b66c55-74sdf"
Jul 14 14:41:15.037933 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:15.037900    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d3edc5b-855e-46ba-9585-55ee5665fa97-tls-cert\") pod \"authorino-7b68b66c55-74sdf\" (UID: \"5d3edc5b-855e-46ba-9585-55ee5665fa97\") " pod="kuadrant-system/authorino-7b68b66c55-74sdf"
Jul 14 14:41:15.038082 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:15.037951    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmg8j\" (UniqueName: \"kubernetes.io/projected/5d3edc5b-855e-46ba-9585-55ee5665fa97-kube-api-access-gmg8j\") pod \"authorino-7b68b66c55-74sdf\" (UID: \"5d3edc5b-855e-46ba-9585-55ee5665fa97\") " pod="kuadrant-system/authorino-7b68b66c55-74sdf"
Jul 14 14:41:15.040437 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:15.040400    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d3edc5b-855e-46ba-9585-55ee5665fa97-tls-cert\") pod \"authorino-7b68b66c55-74sdf\" (UID: \"5d3edc5b-855e-46ba-9585-55ee5665fa97\") " pod="kuadrant-system/authorino-7b68b66c55-74sdf"
Jul 14 14:41:15.047608 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:15.047582    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmg8j\" (UniqueName: \"kubernetes.io/projected/5d3edc5b-855e-46ba-9585-55ee5665fa97-kube-api-access-gmg8j\") pod \"authorino-7b68b66c55-74sdf\" (UID: \"5d3edc5b-855e-46ba-9585-55ee5665fa97\") " pod="kuadrant-system/authorino-7b68b66c55-74sdf"
Jul 14 14:41:15.158617 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:15.158536    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b68b66c55-74sdf"
Jul 14 14:41:15.305456 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:15.305425    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b68b66c55-74sdf"]
Jul 14 14:41:15.307730 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:41:15.307707    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d3edc5b_855e_46ba_9585_55ee5665fa97.slice/crio-f5e900609e4b9c58102fc72af7bc1cfff6081a3bef5a9e19400d897570e6b87b WatchSource:0}: Error finding container f5e900609e4b9c58102fc72af7bc1cfff6081a3bef5a9e19400d897570e6b87b: Status 404 returned error can't find the container with id f5e900609e4b9c58102fc72af7bc1cfff6081a3bef5a9e19400d897570e6b87b
Jul 14 14:41:15.309036 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:15.309021    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 14:41:15.806496 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:15.806461    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b68b66c55-74sdf" event={"ID":"5d3edc5b-855e-46ba-9585-55ee5665fa97","Type":"ContainerStarted","Data":"f5e900609e4b9c58102fc72af7bc1cfff6081a3bef5a9e19400d897570e6b87b"}
Jul 14 14:41:16.814044 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:16.814004    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b68b66c55-74sdf" event={"ID":"5d3edc5b-855e-46ba-9585-55ee5665fa97","Type":"ContainerStarted","Data":"e3f04700aa59555da7238b7f412f333e7d85c46632757a7f29bf24ccae9fa624"}
Jul 14 14:41:16.863656 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:41:16.863605    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b68b66c55-74sdf" podStartSLOduration=2.357715732 podStartE2EDuration="2.863589251s" totalImagesPullingTime="505.873519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:41:16.862138633 +0000 UTC m=+654.528325054" watchObservedRunningTime="2026-07-14 14:41:16.863589251 +0000 UTC m=+654.529775672"
Jul 14 14:42:29.001979 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:29.001886    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5898bfff6b-z28q4"]
Jul 14 14:42:29.002496 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:29.002154    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5898bfff6b-z28q4" podUID="348b64ef-bcd5-4eb5-98e4-f30f812d7251" containerName="manager" containerID="cri-o://f1f6ab896d7acc699806c7d2b7cfd4b6a107cb499f1bafb8da067723b09c2ce1" gracePeriod=30
Jul 14 14:42:29.158878 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:29.158846    2586 generic.go:356] "Generic (PLEG): container finished" podID="348b64ef-bcd5-4eb5-98e4-f30f812d7251" containerID="f1f6ab896d7acc699806c7d2b7cfd4b6a107cb499f1bafb8da067723b09c2ce1" exitCode=0
Jul 14 14:42:29.159008 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:29.158926    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5898bfff6b-z28q4" event={"ID":"348b64ef-bcd5-4eb5-98e4-f30f812d7251","Type":"ContainerDied","Data":"f1f6ab896d7acc699806c7d2b7cfd4b6a107cb499f1bafb8da067723b09c2ce1"}
Jul 14 14:42:29.261641 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:29.261577    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5898bfff6b-z28q4"
Jul 14 14:42:29.333019 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:29.332990    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpd6x\" (UniqueName: \"kubernetes.io/projected/348b64ef-bcd5-4eb5-98e4-f30f812d7251-kube-api-access-fpd6x\") pod \"348b64ef-bcd5-4eb5-98e4-f30f812d7251\" (UID: \"348b64ef-bcd5-4eb5-98e4-f30f812d7251\") "
Jul 14 14:42:29.333196 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:29.333082    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/348b64ef-bcd5-4eb5-98e4-f30f812d7251-webhook-cert\") pod \"348b64ef-bcd5-4eb5-98e4-f30f812d7251\" (UID: \"348b64ef-bcd5-4eb5-98e4-f30f812d7251\") "
Jul 14 14:42:29.335148 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:29.335120    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348b64ef-bcd5-4eb5-98e4-f30f812d7251-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "348b64ef-bcd5-4eb5-98e4-f30f812d7251" (UID: "348b64ef-bcd5-4eb5-98e4-f30f812d7251"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:42:29.335265 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:29.335190    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348b64ef-bcd5-4eb5-98e4-f30f812d7251-kube-api-access-fpd6x" (OuterVolumeSpecName: "kube-api-access-fpd6x") pod "348b64ef-bcd5-4eb5-98e4-f30f812d7251" (UID: "348b64ef-bcd5-4eb5-98e4-f30f812d7251"). InnerVolumeSpecName "kube-api-access-fpd6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:42:29.434605 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:29.434561    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpd6x\" (UniqueName: \"kubernetes.io/projected/348b64ef-bcd5-4eb5-98e4-f30f812d7251-kube-api-access-fpd6x\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:42:29.434605 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:29.434592    2586 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/348b64ef-bcd5-4eb5-98e4-f30f812d7251-webhook-cert\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:42:30.164750 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.164708    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5898bfff6b-z28q4" event={"ID":"348b64ef-bcd5-4eb5-98e4-f30f812d7251","Type":"ContainerDied","Data":"a8f3462e2b95c11059d0e486d6d8d9fc74a61b3a0aa2bccc2c4901cc4c3ad7e0"}
Jul 14 14:42:30.165194 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.164774    2586 scope.go:117] "RemoveContainer" containerID="f1f6ab896d7acc699806c7d2b7cfd4b6a107cb499f1bafb8da067723b09c2ce1"
Jul 14 14:42:30.165194 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.164722    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5898bfff6b-z28q4"
Jul 14 14:42:30.186615 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.186585    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5898bfff6b-z28q4"]
Jul 14 14:42:30.190442 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.190400    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5898bfff6b-z28q4"]
Jul 14 14:42:30.875742 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.875709    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="348b64ef-bcd5-4eb5-98e4-f30f812d7251" path="/var/lib/kubelet/pods/348b64ef-bcd5-4eb5-98e4-f30f812d7251/volumes"
Jul 14 14:42:30.971224 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.971183    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5898bfff6b-r5frs"]
Jul 14 14:42:30.971632 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.971617    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348b64ef-bcd5-4eb5-98e4-f30f812d7251" containerName="manager"
Jul 14 14:42:30.971632 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.971632    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="348b64ef-bcd5-4eb5-98e4-f30f812d7251" containerName="manager"
Jul 14 14:42:30.971765 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.971701    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="348b64ef-bcd5-4eb5-98e4-f30f812d7251" containerName="manager"
Jul 14 14:42:30.976070 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.976052    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5898bfff6b-r5frs"
Jul 14 14:42:30.980119 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.980074    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 14 14:42:30.980119 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.980082    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 14 14:42:30.980307 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.980086    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sn8zt\""
Jul 14 14:42:30.986822 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:30.986795    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5898bfff6b-r5frs"]
Jul 14 14:42:31.049220 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:31.049186    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbd9157b-90f9-4527-afc4-56feb96a2268-webhook-cert\") pod \"maas-controller-5898bfff6b-r5frs\" (UID: \"fbd9157b-90f9-4527-afc4-56feb96a2268\") " pod="opendatahub/maas-controller-5898bfff6b-r5frs"
Jul 14 14:42:31.049394 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:31.049302    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsghw\" (UniqueName: \"kubernetes.io/projected/fbd9157b-90f9-4527-afc4-56feb96a2268-kube-api-access-qsghw\") pod \"maas-controller-5898bfff6b-r5frs\" (UID: \"fbd9157b-90f9-4527-afc4-56feb96a2268\") " pod="opendatahub/maas-controller-5898bfff6b-r5frs"
Jul 14 14:42:31.150474 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:31.150357    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsghw\" (UniqueName: \"kubernetes.io/projected/fbd9157b-90f9-4527-afc4-56feb96a2268-kube-api-access-qsghw\") pod \"maas-controller-5898bfff6b-r5frs\" (UID: \"fbd9157b-90f9-4527-afc4-56feb96a2268\") " pod="opendatahub/maas-controller-5898bfff6b-r5frs"
Jul 14 14:42:31.150474 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:31.150442    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbd9157b-90f9-4527-afc4-56feb96a2268-webhook-cert\") pod \"maas-controller-5898bfff6b-r5frs\" (UID: \"fbd9157b-90f9-4527-afc4-56feb96a2268\") " pod="opendatahub/maas-controller-5898bfff6b-r5frs"
Jul 14 14:42:31.153021 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:31.152984    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbd9157b-90f9-4527-afc4-56feb96a2268-webhook-cert\") pod \"maas-controller-5898bfff6b-r5frs\" (UID: \"fbd9157b-90f9-4527-afc4-56feb96a2268\") " pod="opendatahub/maas-controller-5898bfff6b-r5frs"
Jul 14 14:42:31.166316 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:31.166135    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsghw\" (UniqueName: \"kubernetes.io/projected/fbd9157b-90f9-4527-afc4-56feb96a2268-kube-api-access-qsghw\") pod \"maas-controller-5898bfff6b-r5frs\" (UID: \"fbd9157b-90f9-4527-afc4-56feb96a2268\") " pod="opendatahub/maas-controller-5898bfff6b-r5frs"
Jul 14 14:42:31.288127 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:31.288088    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5898bfff6b-r5frs"
Jul 14 14:42:31.426369 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:31.426342    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5898bfff6b-r5frs"]
Jul 14 14:42:31.428428 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:42:31.428386    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd9157b_90f9_4527_afc4_56feb96a2268.slice/crio-c05a0b3b437574254c4e3405860f3a40c15b90ec7b419fbda0081512e083cc3d WatchSource:0}: Error finding container c05a0b3b437574254c4e3405860f3a40c15b90ec7b419fbda0081512e083cc3d: Status 404 returned error can't find the container with id c05a0b3b437574254c4e3405860f3a40c15b90ec7b419fbda0081512e083cc3d
Jul 14 14:42:32.177010 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:32.176978    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5898bfff6b-r5frs" event={"ID":"fbd9157b-90f9-4527-afc4-56feb96a2268","Type":"ContainerStarted","Data":"2a29a42ad156eeec9f6b0b0cccd2fa1500814f4c195c5504c7c75ab4a43462be"}
Jul 14 14:42:32.177389 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:32.177028    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5898bfff6b-r5frs" event={"ID":"fbd9157b-90f9-4527-afc4-56feb96a2268","Type":"ContainerStarted","Data":"c05a0b3b437574254c4e3405860f3a40c15b90ec7b419fbda0081512e083cc3d"}
Jul 14 14:42:32.177389 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:32.177060    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5898bfff6b-r5frs"
Jul 14 14:42:32.195642 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:32.195591    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5898bfff6b-r5frs" podStartSLOduration=1.7809660219999999 podStartE2EDuration="2.19557465s" totalImagesPullingTime="414.608628ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:42:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:42:32.193677796 +0000 UTC m=+729.859864217" watchObservedRunningTime="2026-07-14 14:42:32.19557465 +0000 UTC m=+729.861761106"
Jul 14 14:42:43.184665 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:42:43.184628    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5898bfff6b-r5frs"
Jul 14 14:45:00.140303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:00.140265    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734005-bwdwp"]
Jul 14 14:45:00.143777 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:00.143760    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734005-bwdwp"
Jul 14 14:45:00.146454 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:00.146429    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5695q\""
Jul 14 14:45:00.153516 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:00.153500    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734005-bwdwp"
Jul 14 14:45:00.161320 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:00.161293    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734005-bwdwp"]
Jul 14 14:45:00.304944 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:00.304794    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734005-bwdwp"]
Jul 14 14:45:00.308192 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:45:00.308156    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af5f455_5199_41c1_9f91_349ef3898d0c.slice/crio-09b000927a4500bc9b52524219dbb59f5917627942e42aa4bbd38e25154d895f WatchSource:0}: Error finding container 09b000927a4500bc9b52524219dbb59f5917627942e42aa4bbd38e25154d895f: Status 404 returned error can't find the container with id 09b000927a4500bc9b52524219dbb59f5917627942e42aa4bbd38e25154d895f
Jul 14 14:45:00.898672 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:00.898629    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734005-bwdwp" event={"ID":"6af5f455-5199-41c1-9f91-349ef3898d0c","Type":"ContainerStarted","Data":"09b000927a4500bc9b52524219dbb59f5917627942e42aa4bbd38e25154d895f"}
Jul 14 14:45:04.921307 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:04.921263    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734005-bwdwp" event={"ID":"6af5f455-5199-41c1-9f91-349ef3898d0c","Type":"ContainerStarted","Data":"9779fab8113a055a8ec191ae8090f257dc6917f34e051f5c9653198af5da165f"}
Jul 14 14:45:04.937903 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:04.937850    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734005-bwdwp" podStartSLOduration=1.263597125 podStartE2EDuration="4.937834255s" totalImagesPullingTime="3.67423713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:45:04.936195492 +0000 UTC m=+882.602381914" watchObservedRunningTime="2026-07-14 14:45:04.937834255 +0000 UTC m=+882.604020676"
Jul 14 14:45:05.928401 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:05.928364    2586 generic.go:356] "Generic (PLEG): container finished" podID="6af5f455-5199-41c1-9f91-349ef3898d0c" containerID="9779fab8113a055a8ec191ae8090f257dc6917f34e051f5c9653198af5da165f" exitCode=0
Jul 14 14:45:05.928814 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:05.928456    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734005-bwdwp" event={"ID":"6af5f455-5199-41c1-9f91-349ef3898d0c","Type":"ContainerDied","Data":"9779fab8113a055a8ec191ae8090f257dc6917f34e051f5c9653198af5da165f"}
Jul 14 14:45:07.076303 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:07.076272    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734005-bwdwp"
Jul 14 14:45:07.940601 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:07.940565    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734005-bwdwp" event={"ID":"6af5f455-5199-41c1-9f91-349ef3898d0c","Type":"ContainerDied","Data":"09b000927a4500bc9b52524219dbb59f5917627942e42aa4bbd38e25154d895f"}
Jul 14 14:45:07.940601 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:07.940597    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734005-bwdwp"
Jul 14 14:45:07.940815 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:07.940619    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09b000927a4500bc9b52524219dbb59f5917627942e42aa4bbd38e25154d895f"
Jul 14 14:45:22.833672 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:22.833585    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:45:22.838001 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:45:22.837980    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:47:26.261103 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:26.261048    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"]
Jul 14 14:47:26.263859 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:26.261305    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n" podUID="214dec35-0b45-4763-8a01-cb485ace1243" containerName="manager" containerID="cri-o://b70525b48481b2bd00de607f9be1f1c0c2a84f96b22565edfd9366aff762ddcf" gracePeriod=10
Jul 14 14:47:29.917047 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:29.917021    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"
Jul 14 14:47:29.964312 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:29.964231    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7vhl\" (UniqueName: \"kubernetes.io/projected/214dec35-0b45-4763-8a01-cb485ace1243-kube-api-access-j7vhl\") pod \"214dec35-0b45-4763-8a01-cb485ace1243\" (UID: \"214dec35-0b45-4763-8a01-cb485ace1243\") "
Jul 14 14:47:29.964312 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:29.964289    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/214dec35-0b45-4763-8a01-cb485ace1243-extensions-socket-volume\") pod \"214dec35-0b45-4763-8a01-cb485ace1243\" (UID: \"214dec35-0b45-4763-8a01-cb485ace1243\") "
Jul 14 14:47:29.964983 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:29.964686    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/214dec35-0b45-4763-8a01-cb485ace1243-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "214dec35-0b45-4763-8a01-cb485ace1243" (UID: "214dec35-0b45-4763-8a01-cb485ace1243"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:47:29.965099 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:29.965066    2586 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/214dec35-0b45-4763-8a01-cb485ace1243-extensions-socket-volume\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:47:29.966568 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:29.966546    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214dec35-0b45-4763-8a01-cb485ace1243-kube-api-access-j7vhl" (OuterVolumeSpecName: "kube-api-access-j7vhl") pod "214dec35-0b45-4763-8a01-cb485ace1243" (UID: "214dec35-0b45-4763-8a01-cb485ace1243"). InnerVolumeSpecName "kube-api-access-j7vhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:47:30.065793 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:30.065760    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7vhl\" (UniqueName: \"kubernetes.io/projected/214dec35-0b45-4763-8a01-cb485ace1243-kube-api-access-j7vhl\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:47:30.642851 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:30.642817    2586 generic.go:356] "Generic (PLEG): container finished" podID="214dec35-0b45-4763-8a01-cb485ace1243" containerID="b70525b48481b2bd00de607f9be1f1c0c2a84f96b22565edfd9366aff762ddcf" exitCode=0
Jul 14 14:47:30.643025 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:30.642862    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"
Jul 14 14:47:30.643025 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:30.642890    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n" event={"ID":"214dec35-0b45-4763-8a01-cb485ace1243","Type":"ContainerDied","Data":"b70525b48481b2bd00de607f9be1f1c0c2a84f96b22565edfd9366aff762ddcf"}
Jul 14 14:47:30.643025 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:30.642940    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n" event={"ID":"214dec35-0b45-4763-8a01-cb485ace1243","Type":"ContainerDied","Data":"5a77bdbf4618cfd5cdfedb19eb69364d3fb1016cca6b4fb99f0de55a879e4d58"}
Jul 14 14:47:30.643025 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:30.642960    2586 scope.go:117] "RemoveContainer" containerID="b70525b48481b2bd00de607f9be1f1c0c2a84f96b22565edfd9366aff762ddcf"
Jul 14 14:47:30.653510 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:30.653489    2586 scope.go:117] "RemoveContainer" containerID="b70525b48481b2bd00de607f9be1f1c0c2a84f96b22565edfd9366aff762ddcf"
Jul 14 14:47:30.653755 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:47:30.653727    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70525b48481b2bd00de607f9be1f1c0c2a84f96b22565edfd9366aff762ddcf\": container with ID starting with b70525b48481b2bd00de607f9be1f1c0c2a84f96b22565edfd9366aff762ddcf not found: ID does not exist" containerID="b70525b48481b2bd00de607f9be1f1c0c2a84f96b22565edfd9366aff762ddcf"
Jul 14 14:47:30.653803 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:30.653763    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70525b48481b2bd00de607f9be1f1c0c2a84f96b22565edfd9366aff762ddcf"} err="failed to get container status \"b70525b48481b2bd00de607f9be1f1c0c2a84f96b22565edfd9366aff762ddcf\": rpc error: code = NotFound desc = could not find container \"b70525b48481b2bd00de607f9be1f1c0c2a84f96b22565edfd9366aff762ddcf\": container with ID starting with b70525b48481b2bd00de607f9be1f1c0c2a84f96b22565edfd9366aff762ddcf not found: ID does not exist"
Jul 14 14:47:30.664936 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:30.664913    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"]
Jul 14 14:47:30.668689 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:30.668669    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lhh2n"]
Jul 14 14:47:30.880407 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:47:30.880374    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="214dec35-0b45-4763-8a01-cb485ace1243" path="/var/lib/kubelet/pods/214dec35-0b45-4763-8a01-cb485ace1243/volumes"
Jul 14 14:48:27.444949 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.444918    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp"]
Jul 14 14:48:27.445336 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.445273    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="214dec35-0b45-4763-8a01-cb485ace1243" containerName="manager"
Jul 14 14:48:27.445336 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.445283    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="214dec35-0b45-4763-8a01-cb485ace1243" containerName="manager"
Jul 14 14:48:27.445336 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.445305    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6af5f455-5199-41c1-9f91-349ef3898d0c" containerName="cleanup"
Jul 14 14:48:27.445336 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.445312    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6af5f455-5199-41c1-9f91-349ef3898d0c" containerName="cleanup"
Jul 14 14:48:27.445486 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.445366    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6af5f455-5199-41c1-9f91-349ef3898d0c" containerName="cleanup"
Jul 14 14:48:27.445486 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.445373    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="214dec35-0b45-4763-8a01-cb485ace1243" containerName="manager"
Jul 14 14:48:27.448696 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.448681    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp"
Jul 14 14:48:27.451451 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.451400    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fzsbb\""
Jul 14 14:48:27.457382 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.457359    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp"]
Jul 14 14:48:27.480393 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.480364    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wvwx\" (UniqueName: \"kubernetes.io/projected/318d09e6-2ac2-42e1-beb2-99f961d7e886-kube-api-access-5wvwx\") pod \"kuadrant-operator-controller-manager-86878c769f-dx7fp\" (UID: \"318d09e6-2ac2-42e1-beb2-99f961d7e886\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp"
Jul 14 14:48:27.480521 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.480504    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/318d09e6-2ac2-42e1-beb2-99f961d7e886-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dx7fp\" (UID: \"318d09e6-2ac2-42e1-beb2-99f961d7e886\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp"
Jul 14 14:48:27.581055 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.581025    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/318d09e6-2ac2-42e1-beb2-99f961d7e886-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dx7fp\" (UID: \"318d09e6-2ac2-42e1-beb2-99f961d7e886\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp"
Jul 14 14:48:27.581254 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.581084    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wvwx\" (UniqueName: \"kubernetes.io/projected/318d09e6-2ac2-42e1-beb2-99f961d7e886-kube-api-access-5wvwx\") pod \"kuadrant-operator-controller-manager-86878c769f-dx7fp\" (UID: \"318d09e6-2ac2-42e1-beb2-99f961d7e886\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp"
Jul 14 14:48:27.581485 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.581462    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/318d09e6-2ac2-42e1-beb2-99f961d7e886-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dx7fp\" (UID: \"318d09e6-2ac2-42e1-beb2-99f961d7e886\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp"
Jul 14 14:48:27.605897 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.605864    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wvwx\" (UniqueName: \"kubernetes.io/projected/318d09e6-2ac2-42e1-beb2-99f961d7e886-kube-api-access-5wvwx\") pod \"kuadrant-operator-controller-manager-86878c769f-dx7fp\" (UID: \"318d09e6-2ac2-42e1-beb2-99f961d7e886\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp"
Jul 14 14:48:27.759669 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.759577    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp"
Jul 14 14:48:27.899372 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:27.899336    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp"]
Jul 14 14:48:27.902674 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:48:27.902647    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318d09e6_2ac2_42e1_beb2_99f961d7e886.slice/crio-fcf6c1f15780669c7050ad5d1e92a89a87d42039fed028d0835166ebf8d4e201 WatchSource:0}: Error finding container fcf6c1f15780669c7050ad5d1e92a89a87d42039fed028d0835166ebf8d4e201: Status 404 returned error can't find the container with id fcf6c1f15780669c7050ad5d1e92a89a87d42039fed028d0835166ebf8d4e201
Jul 14 14:48:28.915225 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:28.915173    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp" event={"ID":"318d09e6-2ac2-42e1-beb2-99f961d7e886","Type":"ContainerStarted","Data":"fb17258410f38345208d3813e8e0435ba341eaab8e23b5c678f896194cea6d2c"}
Jul 14 14:48:28.915640 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:28.915246    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp" event={"ID":"318d09e6-2ac2-42e1-beb2-99f961d7e886","Type":"ContainerStarted","Data":"fcf6c1f15780669c7050ad5d1e92a89a87d42039fed028d0835166ebf8d4e201"}
Jul 14 14:48:28.915640 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:28.915337    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp"
Jul 14 14:48:28.935997 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:28.935945    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp" podStartSLOduration=1.935929676 podStartE2EDuration="1.935929676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:48:28.93383999 +0000 UTC m=+1086.600026409" watchObservedRunningTime="2026-07-14 14:48:28.935929676 +0000 UTC m=+1086.602116098"
Jul 14 14:48:39.922742 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:48:39.922708    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dx7fp"
Jul 14 14:50:22.865492 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:50:22.865459    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:50:22.867497 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:50:22.867478    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:54:58.719457 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:58.719391    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"]
Jul 14 14:54:58.723212 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:58.723189    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:54:58.726117 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:58.726093    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2b2b03f7-serving-cert\""
Jul 14 14:54:58.734966 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:58.734945    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"]
Jul 14 14:54:58.836103 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:58.836070    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-openshift-service-ca\") pod \"maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq\" (UID: \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:54:58.836103 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:58.836107    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlktb\" (UniqueName: \"kubernetes.io/projected/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-kube-api-access-mlktb\") pod \"maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq\" (UID: \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:54:58.836297 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:58.836135    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-maas-api-tls\") pod \"maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq\" (UID: \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:54:58.937301 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:58.937261    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-openshift-service-ca\") pod \"maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq\" (UID: \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:54:58.937301 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:58.937300    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlktb\" (UniqueName: \"kubernetes.io/projected/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-kube-api-access-mlktb\") pod \"maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq\" (UID: \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:54:58.937580 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:58.937331    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-maas-api-tls\") pod \"maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq\" (UID: \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:54:58.937580 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:54:58.937449    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-serving-cert: secret "maas-api-e2e-ait-2b2b03f7-serving-cert" not found
Jul 14 14:54:58.937580 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:54:58.937530    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-maas-api-tls podName:981c2fc5-3c6d-4f78-b600-ee75990e3c9b nodeName:}" failed. No retries permitted until 2026-07-14 14:54:59.437511765 +0000 UTC m=+1477.103698165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-maas-api-tls") pod "maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq" (UID: "981c2fc5-3c6d-4f78-b600-ee75990e3c9b") : secret "maas-api-e2e-ait-2b2b03f7-serving-cert" not found
Jul 14 14:54:58.937925 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:58.937903    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-openshift-service-ca\") pod \"maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq\" (UID: \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:54:58.950636 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:58.950606    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlktb\" (UniqueName: \"kubernetes.io/projected/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-kube-api-access-mlktb\") pod \"maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq\" (UID: \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:54:59.443474 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:59.443440    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-maas-api-tls\") pod \"maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq\" (UID: \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:54:59.446140 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:59.446110    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-maas-api-tls\") pod \"maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq\" (UID: \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:54:59.634398 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:59.634364    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:54:59.781010 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:59.780974    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"]
Jul 14 14:54:59.783256 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:54:59.783227    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981c2fc5_3c6d_4f78_b600_ee75990e3c9b.slice/crio-5baa8df2f9c27b58321f6ec9e091402ff61893d7fcb837ca923693bdd69d5144 WatchSource:0}: Error finding container 5baa8df2f9c27b58321f6ec9e091402ff61893d7fcb837ca923693bdd69d5144: Status 404 returned error can't find the container with id 5baa8df2f9c27b58321f6ec9e091402ff61893d7fcb837ca923693bdd69d5144
Jul 14 14:54:59.784664 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:59.784645    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 14:54:59.809878 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:54:59.809843    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq" event={"ID":"981c2fc5-3c6d-4f78-b600-ee75990e3c9b","Type":"ContainerStarted","Data":"5baa8df2f9c27b58321f6ec9e091402ff61893d7fcb837ca923693bdd69d5144"}
Jul 14 14:55:00.816321 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:00.816265    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq" event={"ID":"981c2fc5-3c6d-4f78-b600-ee75990e3c9b","Type":"ContainerStarted","Data":"4125dcad9916ca0ad2c77a8da91ead329e48c5c9878cd13540051404e722e278"}
Jul 14 14:55:00.816781 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:00.816468    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:55:00.836634 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:00.836583    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq" podStartSLOduration=2.2678021 podStartE2EDuration="2.836563401s" totalImagesPullingTime="568.761301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:55:00.833361759 +0000 UTC m=+1478.499548181" watchObservedRunningTime="2026-07-14 14:55:00.836563401 +0000 UTC m=+1478.502749822"
Jul 14 14:55:03.677071 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:03.677034    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h"]
Jul 14 14:55:03.681047 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:03.681022    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h"
Jul 14 14:55:03.684017 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:03.683990    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5695q\""
Jul 14 14:55:03.689678 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:03.689606    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h"]
Jul 14 14:55:03.779604 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:03.779567    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e531b8-c99b-4984-befe-e3eea47e27ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h\" (UID: \"e5e531b8-c99b-4984-befe-e3eea47e27ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h"
Jul 14 14:55:03.880355 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:03.880313    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e531b8-c99b-4984-befe-e3eea47e27ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h\" (UID: \"e5e531b8-c99b-4984-befe-e3eea47e27ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h"
Jul 14 14:55:03.880985 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:03.880964    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e531b8-c99b-4984-befe-e3eea47e27ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h\" (UID: \"e5e531b8-c99b-4984-befe-e3eea47e27ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h"
Jul 14 14:55:03.995620 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:03.995525    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h"
Jul 14 14:55:04.141405 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:04.141367    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h"]
Jul 14 14:55:04.144603 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:55:04.144575    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e531b8_c99b_4984_befe_e3eea47e27ee.slice/crio-147f11da353d5c374107810daebe9312e8b91b6ef45b555168badad2274031c8 WatchSource:0}: Error finding container 147f11da353d5c374107810daebe9312e8b91b6ef45b555168badad2274031c8: Status 404 returned error can't find the container with id 147f11da353d5c374107810daebe9312e8b91b6ef45b555168badad2274031c8
Jul 14 14:55:04.837142 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:04.837105    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h" event={"ID":"e5e531b8-c99b-4984-befe-e3eea47e27ee","Type":"ContainerStarted","Data":"9579354aeed5b454ce2783087b1dc0b22206cf4b5cce33674dfc39b12978580f"}
Jul 14 14:55:04.837636 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:04.837165    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h" event={"ID":"e5e531b8-c99b-4984-befe-e3eea47e27ee","Type":"ContainerStarted","Data":"147f11da353d5c374107810daebe9312e8b91b6ef45b555168badad2274031c8"}
Jul 14 14:55:04.854042 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:04.853987    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h" podStartSLOduration=1.853969689 podStartE2EDuration="1.853969689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:55:04.85263026 +0000 UTC m=+1482.518816681" watchObservedRunningTime="2026-07-14 14:55:04.853969689 +0000 UTC m=+1482.520156110"
Jul 14 14:55:05.843593 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:05.843553    2586 generic.go:356] "Generic (PLEG): container finished" podID="e5e531b8-c99b-4984-befe-e3eea47e27ee" containerID="9579354aeed5b454ce2783087b1dc0b22206cf4b5cce33674dfc39b12978580f" exitCode=7
Jul 14 14:55:05.843984 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:05.843638    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h" event={"ID":"e5e531b8-c99b-4984-befe-e3eea47e27ee","Type":"ContainerDied","Data":"9579354aeed5b454ce2783087b1dc0b22206cf4b5cce33674dfc39b12978580f"}
Jul 14 14:55:05.843984 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:05.843827    2586 scope.go:117] "RemoveContainer" containerID="9579354aeed5b454ce2783087b1dc0b22206cf4b5cce33674dfc39b12978580f"
Jul 14 14:55:06.827596 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:06.827569    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:55:06.851556 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:06.851440    2586 generic.go:356] "Generic (PLEG): container finished" podID="e5e531b8-c99b-4984-befe-e3eea47e27ee" containerID="ad0e66754ce36418586a440e1ad0bafa6fd052d619804ad4b3711c93b138fd64" exitCode=7
Jul 14 14:55:06.851556 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:06.851508    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h" event={"ID":"e5e531b8-c99b-4984-befe-e3eea47e27ee","Type":"ContainerDied","Data":"ad0e66754ce36418586a440e1ad0bafa6fd052d619804ad4b3711c93b138fd64"}
Jul 14 14:55:06.852030 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:06.851573    2586 scope.go:117] "RemoveContainer" containerID="9579354aeed5b454ce2783087b1dc0b22206cf4b5cce33674dfc39b12978580f"
Jul 14 14:55:06.852030 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:06.851768    2586 scope.go:117] "RemoveContainer" containerID="ad0e66754ce36418586a440e1ad0bafa6fd052d619804ad4b3711c93b138fd64"
Jul 14 14:55:06.852030 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:55:06.852008    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h_odh-ai-gateway-infra(e5e531b8-c99b-4984-befe-e3eea47e27ee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h" podUID="e5e531b8-c99b-4984-befe-e3eea47e27ee"
Jul 14 14:55:07.858406 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:07.858375    2586 scope.go:117] "RemoveContainer" containerID="ad0e66754ce36418586a440e1ad0bafa6fd052d619804ad4b3711c93b138fd64"
Jul 14 14:55:07.858880 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:55:07.858657    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h_odh-ai-gateway-infra(e5e531b8-c99b-4984-befe-e3eea47e27ee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h" podUID="e5e531b8-c99b-4984-befe-e3eea47e27ee"
Jul 14 14:55:20.871102 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:20.871062    2586 scope.go:117] "RemoveContainer" containerID="ad0e66754ce36418586a440e1ad0bafa6fd052d619804ad4b3711c93b138fd64"
Jul 14 14:55:21.925679 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:21.925648    2586 generic.go:356] "Generic (PLEG): container finished" podID="e5e531b8-c99b-4984-befe-e3eea47e27ee" containerID="6c375a0a27f05854933c5d6f9bdd60db3809cc7664cfc8ba6cf037ee1d4e93ad" exitCode=0
Jul 14 14:55:21.926048 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:21.925721    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h" event={"ID":"e5e531b8-c99b-4984-befe-e3eea47e27ee","Type":"ContainerDied","Data":"6c375a0a27f05854933c5d6f9bdd60db3809cc7664cfc8ba6cf037ee1d4e93ad"}
Jul 14 14:55:21.926048 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:21.925780    2586 scope.go:117] "RemoveContainer" containerID="ad0e66754ce36418586a440e1ad0bafa6fd052d619804ad4b3711c93b138fd64"
Jul 14 14:55:22.899924 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:22.899900    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:55:22.900817 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:22.900798    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 14:55:22.953754 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:22.953726    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h"]
Jul 14 14:55:23.069877 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:23.069854    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h"
Jul 14 14:55:23.159328 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:23.159230    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e531b8-c99b-4984-befe-e3eea47e27ee-maas-api-service-ca\") pod \"e5e531b8-c99b-4984-befe-e3eea47e27ee\" (UID: \"e5e531b8-c99b-4984-befe-e3eea47e27ee\") "
Jul 14 14:55:23.159635 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:23.159613    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e531b8-c99b-4984-befe-e3eea47e27ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5e531b8-c99b-4984-befe-e3eea47e27ee" (UID: "e5e531b8-c99b-4984-befe-e3eea47e27ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:55:23.260489 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:23.260453    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e531b8-c99b-4984-befe-e3eea47e27ee-maas-api-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:55:23.939593 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:23.939558    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h" event={"ID":"e5e531b8-c99b-4984-befe-e3eea47e27ee","Type":"ContainerDied","Data":"147f11da353d5c374107810daebe9312e8b91b6ef45b555168badad2274031c8"}
Jul 14 14:55:23.939593 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:23.939588    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h"
Jul 14 14:55:23.939807 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:23.939605    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="147f11da353d5c374107810daebe9312e8b91b6ef45b555168badad2274031c8"
Jul 14 14:55:23.970663 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:23.970630    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h"]
Jul 14 14:55:23.976661 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:23.976634    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-4vs6h"]
Jul 14 14:55:24.876125 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:55:24.876092    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e531b8-c99b-4984-befe-e3eea47e27ee" path="/var/lib/kubelet/pods/e5e531b8-c99b-4984-befe-e3eea47e27ee/volumes"
Jul 14 14:56:03.823699 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:03.823609    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw"]
Jul 14 14:56:03.824152 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:03.824059    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e531b8-c99b-4984-befe-e3eea47e27ee" containerName="revoke-keys"
Jul 14 14:56:03.824152 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:03.824072    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e531b8-c99b-4984-befe-e3eea47e27ee" containerName="revoke-keys"
Jul 14 14:56:03.824152 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:03.824081    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e531b8-c99b-4984-befe-e3eea47e27ee" containerName="revoke-keys"
Jul 14 14:56:03.824152 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:03.824087    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e531b8-c99b-4984-befe-e3eea47e27ee" containerName="revoke-keys"
Jul 14 14:56:03.824152 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:03.824097    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e531b8-c99b-4984-befe-e3eea47e27ee" containerName="revoke-keys"
Jul 14 14:56:03.824152 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:03.824103    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e531b8-c99b-4984-befe-e3eea47e27ee" containerName="revoke-keys"
Jul 14 14:56:03.824372 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:03.824163    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e531b8-c99b-4984-befe-e3eea47e27ee" containerName="revoke-keys"
Jul 14 14:56:03.824372 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:03.824176    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e531b8-c99b-4984-befe-e3eea47e27ee" containerName="revoke-keys"
Jul 14 14:56:03.827373 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:03.827356    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw"
Jul 14 14:56:03.830407 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:03.830385    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5695q\""
Jul 14 14:56:03.835720 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:03.835695    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw"]
Jul 14 14:56:03.929079 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:03.929042    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/537ca958-8074-4453-86c5-b2f94eb839bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw\" (UID: \"537ca958-8074-4453-86c5-b2f94eb839bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw"
Jul 14 14:56:04.029956 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:04.029922    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/537ca958-8074-4453-86c5-b2f94eb839bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw\" (UID: \"537ca958-8074-4453-86c5-b2f94eb839bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw"
Jul 14 14:56:04.030591 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:04.030574    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/537ca958-8074-4453-86c5-b2f94eb839bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw\" (UID: \"537ca958-8074-4453-86c5-b2f94eb839bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw"
Jul 14 14:56:04.138952 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:04.138871    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw"
Jul 14 14:56:04.276572 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:04.276550    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw"]
Jul 14 14:56:04.278807 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:56:04.278769    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537ca958_8074_4453_86c5_b2f94eb839bc.slice/crio-445fa8893894ff87661c8a880e49da75f43de6cdbb3ac77ec81876495bb88018 WatchSource:0}: Error finding container 445fa8893894ff87661c8a880e49da75f43de6cdbb3ac77ec81876495bb88018: Status 404 returned error can't find the container with id 445fa8893894ff87661c8a880e49da75f43de6cdbb3ac77ec81876495bb88018
Jul 14 14:56:05.145684 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:05.145653    2586 generic.go:356] "Generic (PLEG): container finished" podID="537ca958-8074-4453-86c5-b2f94eb839bc" containerID="b55bf9fbef34329b9aa059aad9a8a7f6afd5e3f5d5a24db9659c2268f0647c12" exitCode=0
Jul 14 14:56:05.146089 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:05.145747    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw" event={"ID":"537ca958-8074-4453-86c5-b2f94eb839bc","Type":"ContainerDied","Data":"b55bf9fbef34329b9aa059aad9a8a7f6afd5e3f5d5a24db9659c2268f0647c12"}
Jul 14 14:56:05.146089 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:05.145799    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw" event={"ID":"537ca958-8074-4453-86c5-b2f94eb839bc","Type":"ContainerStarted","Data":"445fa8893894ff87661c8a880e49da75f43de6cdbb3ac77ec81876495bb88018"}
Jul 14 14:56:06.279552 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:06.279529    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw"
Jul 14 14:56:06.350397 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:06.350363    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/537ca958-8074-4453-86c5-b2f94eb839bc-maas-api-service-ca\") pod \"537ca958-8074-4453-86c5-b2f94eb839bc\" (UID: \"537ca958-8074-4453-86c5-b2f94eb839bc\") "
Jul 14 14:56:06.350708 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:06.350689    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/537ca958-8074-4453-86c5-b2f94eb839bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "537ca958-8074-4453-86c5-b2f94eb839bc" (UID: "537ca958-8074-4453-86c5-b2f94eb839bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:56:06.451223 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:06.451190    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/537ca958-8074-4453-86c5-b2f94eb839bc-maas-api-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:56:07.156711 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:07.156624    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw" event={"ID":"537ca958-8074-4453-86c5-b2f94eb839bc","Type":"ContainerDied","Data":"445fa8893894ff87661c8a880e49da75f43de6cdbb3ac77ec81876495bb88018"}
Jul 14 14:56:07.156711 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:07.156655    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw"
Jul 14 14:56:07.156711 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:07.156672    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="445fa8893894ff87661c8a880e49da75f43de6cdbb3ac77ec81876495bb88018"
Jul 14 14:56:14.190755 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:14.189999    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"]
Jul 14 14:56:14.190755 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:14.190238    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq" podUID="981c2fc5-3c6d-4f78-b600-ee75990e3c9b" containerName="maas-api" containerID="cri-o://4125dcad9916ca0ad2c77a8da91ead329e48c5c9878cd13540051404e722e278" gracePeriod=30
Jul 14 14:56:14.444885 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:14.444827    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:56:14.524471 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:14.524403    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-openshift-service-ca\") pod \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\" (UID: \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\") "
Jul 14 14:56:14.524665 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:14.524476    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlktb\" (UniqueName: \"kubernetes.io/projected/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-kube-api-access-mlktb\") pod \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\" (UID: \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\") "
Jul 14 14:56:14.524665 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:14.524545    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-maas-api-tls\") pod \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\" (UID: \"981c2fc5-3c6d-4f78-b600-ee75990e3c9b\") "
Jul 14 14:56:14.524790 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:14.524768    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "981c2fc5-3c6d-4f78-b600-ee75990e3c9b" (UID: "981c2fc5-3c6d-4f78-b600-ee75990e3c9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:56:14.526880 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:14.526853    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "981c2fc5-3c6d-4f78-b600-ee75990e3c9b" (UID: "981c2fc5-3c6d-4f78-b600-ee75990e3c9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:56:14.526880 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:14.526857    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-kube-api-access-mlktb" (OuterVolumeSpecName: "kube-api-access-mlktb") pod "981c2fc5-3c6d-4f78-b600-ee75990e3c9b" (UID: "981c2fc5-3c6d-4f78-b600-ee75990e3c9b"). InnerVolumeSpecName "kube-api-access-mlktb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:56:14.625216 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:14.625172    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-maas-api-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:56:14.625216 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:14.625207    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-openshift-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:56:14.625216 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:14.625216    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlktb\" (UniqueName: \"kubernetes.io/projected/981c2fc5-3c6d-4f78-b600-ee75990e3c9b-kube-api-access-mlktb\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:56:15.200026 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:15.199989    2586 generic.go:356] "Generic (PLEG): container finished" podID="981c2fc5-3c6d-4f78-b600-ee75990e3c9b" containerID="4125dcad9916ca0ad2c77a8da91ead329e48c5c9878cd13540051404e722e278" exitCode=0
Jul 14 14:56:15.200529 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:15.200035    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"
Jul 14 14:56:15.200529 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:15.200078    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq" event={"ID":"981c2fc5-3c6d-4f78-b600-ee75990e3c9b","Type":"ContainerDied","Data":"4125dcad9916ca0ad2c77a8da91ead329e48c5c9878cd13540051404e722e278"}
Jul 14 14:56:15.200529 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:15.200139    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq" event={"ID":"981c2fc5-3c6d-4f78-b600-ee75990e3c9b","Type":"ContainerDied","Data":"5baa8df2f9c27b58321f6ec9e091402ff61893d7fcb837ca923693bdd69d5144"}
Jul 14 14:56:15.200529 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:15.200174    2586 scope.go:117] "RemoveContainer" containerID="4125dcad9916ca0ad2c77a8da91ead329e48c5c9878cd13540051404e722e278"
Jul 14 14:56:15.211700 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:15.211672    2586 scope.go:117] "RemoveContainer" containerID="4125dcad9916ca0ad2c77a8da91ead329e48c5c9878cd13540051404e722e278"
Jul 14 14:56:15.211942 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:56:15.211923    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4125dcad9916ca0ad2c77a8da91ead329e48c5c9878cd13540051404e722e278\": container with ID starting with 4125dcad9916ca0ad2c77a8da91ead329e48c5c9878cd13540051404e722e278 not found: ID does not exist" containerID="4125dcad9916ca0ad2c77a8da91ead329e48c5c9878cd13540051404e722e278"
Jul 14 14:56:15.211989 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:15.211957    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4125dcad9916ca0ad2c77a8da91ead329e48c5c9878cd13540051404e722e278"} err="failed to get container status \"4125dcad9916ca0ad2c77a8da91ead329e48c5c9878cd13540051404e722e278\": rpc error: code = NotFound desc = could not find container \"4125dcad9916ca0ad2c77a8da91ead329e48c5c9878cd13540051404e722e278\": container with ID starting with 4125dcad9916ca0ad2c77a8da91ead329e48c5c9878cd13540051404e722e278 not found: ID does not exist"
Jul 14 14:56:15.218614 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:15.218590    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"]
Jul 14 14:56:15.220452 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:15.220430    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b2b03f7-5b5486c459-qdccq"]
Jul 14 14:56:16.876662 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:16.876629    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="981c2fc5-3c6d-4f78-b600-ee75990e3c9b" path="/var/lib/kubelet/pods/981c2fc5-3c6d-4f78-b600-ee75990e3c9b/volumes"
Jul 14 14:56:30.887792 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:30.887762    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"]
Jul 14 14:56:30.888261 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:30.888239    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="537ca958-8074-4453-86c5-b2f94eb839bc" containerName="revoke-keys"
Jul 14 14:56:30.888347 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:30.888286    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="537ca958-8074-4453-86c5-b2f94eb839bc" containerName="revoke-keys"
Jul 14 14:56:30.888347 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:30.888300    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981c2fc5-3c6d-4f78-b600-ee75990e3c9b" containerName="maas-api"
Jul 14 14:56:30.888347 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:30.888305    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="981c2fc5-3c6d-4f78-b600-ee75990e3c9b" containerName="maas-api"
Jul 14 14:56:30.888533 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:30.888394    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="981c2fc5-3c6d-4f78-b600-ee75990e3c9b" containerName="maas-api"
Jul 14 14:56:30.888533 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:30.888428    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e531b8-c99b-4984-befe-e3eea47e27ee" containerName="revoke-keys"
Jul 14 14:56:30.888533 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:30.888439    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="537ca958-8074-4453-86c5-b2f94eb839bc" containerName="revoke-keys"
Jul 14 14:56:30.892926 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:30.892896    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:56:30.895685 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:30.895658    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-11424108-serving-cert\""
Jul 14 14:56:30.902299 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:30.902270    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"]
Jul 14 14:56:30.981105 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:30.981068    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24b96\" (UniqueName: \"kubernetes.io/projected/aee2286c-8dd0-4444-b75e-56fedb771c46-kube-api-access-24b96\") pod \"maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk\" (UID: \"aee2286c-8dd0-4444-b75e-56fedb771c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:56:30.981293 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:30.981177    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee2286c-8dd0-4444-b75e-56fedb771c46-maas-api-tls\") pod \"maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk\" (UID: \"aee2286c-8dd0-4444-b75e-56fedb771c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:56:30.981361 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:30.981340    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee2286c-8dd0-4444-b75e-56fedb771c46-openshift-service-ca\") pod \"maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk\" (UID: \"aee2286c-8dd0-4444-b75e-56fedb771c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:56:31.082372 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:31.082342    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24b96\" (UniqueName: \"kubernetes.io/projected/aee2286c-8dd0-4444-b75e-56fedb771c46-kube-api-access-24b96\") pod \"maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk\" (UID: \"aee2286c-8dd0-4444-b75e-56fedb771c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:56:31.082556 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:31.082387    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee2286c-8dd0-4444-b75e-56fedb771c46-maas-api-tls\") pod \"maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk\" (UID: \"aee2286c-8dd0-4444-b75e-56fedb771c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:56:31.082556 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:31.082476    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee2286c-8dd0-4444-b75e-56fedb771c46-openshift-service-ca\") pod \"maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk\" (UID: \"aee2286c-8dd0-4444-b75e-56fedb771c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:56:31.083275 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:31.083192    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee2286c-8dd0-4444-b75e-56fedb771c46-openshift-service-ca\") pod \"maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk\" (UID: \"aee2286c-8dd0-4444-b75e-56fedb771c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:56:31.085016 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:31.084993    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee2286c-8dd0-4444-b75e-56fedb771c46-maas-api-tls\") pod \"maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk\" (UID: \"aee2286c-8dd0-4444-b75e-56fedb771c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:56:31.092059 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:31.092034    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24b96\" (UniqueName: \"kubernetes.io/projected/aee2286c-8dd0-4444-b75e-56fedb771c46-kube-api-access-24b96\") pod \"maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk\" (UID: \"aee2286c-8dd0-4444-b75e-56fedb771c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:56:31.206016 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:31.205977    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:56:31.348651 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:31.348627    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"]
Jul 14 14:56:31.350648 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:56:31.350621    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee2286c_8dd0_4444_b75e_56fedb771c46.slice/crio-02b21e2049c815aae45d40ba95237ac65867af8c375475d489a0ef2998fd2723 WatchSource:0}: Error finding container 02b21e2049c815aae45d40ba95237ac65867af8c375475d489a0ef2998fd2723: Status 404 returned error can't find the container with id 02b21e2049c815aae45d40ba95237ac65867af8c375475d489a0ef2998fd2723
Jul 14 14:56:32.284025 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:32.283983    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk" event={"ID":"aee2286c-8dd0-4444-b75e-56fedb771c46","Type":"ContainerStarted","Data":"02b21e2049c815aae45d40ba95237ac65867af8c375475d489a0ef2998fd2723"}
Jul 14 14:56:33.291500 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:33.291442    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk" event={"ID":"aee2286c-8dd0-4444-b75e-56fedb771c46","Type":"ContainerStarted","Data":"0a98e1a3fd280d88b17a3ac82a579551a73799f25685b71532f5ac828a5039a8"}
Jul 14 14:56:33.291910 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:33.291526    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:56:33.314626 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:33.314573    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk" podStartSLOduration=2.223384124 podStartE2EDuration="3.314557146s" totalImagesPullingTime="1.091173022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:56:33.312519971 +0000 UTC m=+1570.978706394" watchObservedRunningTime="2026-07-14 14:56:33.314557146 +0000 UTC m=+1570.980743567"
Jul 14 14:56:35.338611 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:35.338573    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77"]
Jul 14 14:56:35.342811 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:35.342793    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77"
Jul 14 14:56:35.345785 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:35.345764    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5695q\""
Jul 14 14:56:35.348672 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:35.348641    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77"]
Jul 14 14:56:35.523865 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:35.523825    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8865865-2484-43fe-b15d-3f62e420f0eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-11424108-dcm77\" (UID: \"f8865865-2484-43fe-b15d-3f62e420f0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77"
Jul 14 14:56:35.625010 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:35.624921    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8865865-2484-43fe-b15d-3f62e420f0eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-11424108-dcm77\" (UID: \"f8865865-2484-43fe-b15d-3f62e420f0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77"
Jul 14 14:56:35.625626 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:35.625594    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8865865-2484-43fe-b15d-3f62e420f0eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-11424108-dcm77\" (UID: \"f8865865-2484-43fe-b15d-3f62e420f0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77"
Jul 14 14:56:35.655008 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:35.654977    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77"
Jul 14 14:56:35.793289 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:35.793264    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77"]
Jul 14 14:56:35.795514 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:56:35.795475    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8865865_2484_43fe_b15d_3f62e420f0eb.slice/crio-9a1783f49936d7ff4b9ef2768f3a2b0ff87283639c47621e4555630ce58a1fbb WatchSource:0}: Error finding container 9a1783f49936d7ff4b9ef2768f3a2b0ff87283639c47621e4555630ce58a1fbb: Status 404 returned error can't find the container with id 9a1783f49936d7ff4b9ef2768f3a2b0ff87283639c47621e4555630ce58a1fbb
Jul 14 14:56:36.309101 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:36.309012    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77" event={"ID":"f8865865-2484-43fe-b15d-3f62e420f0eb","Type":"ContainerStarted","Data":"259b67cb2ea242bdbfa346eca40a1cb8c8d5b4f32e8bcfa3f7e4971db562d6cc"}
Jul 14 14:56:36.309101 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:36.309061    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77" event={"ID":"f8865865-2484-43fe-b15d-3f62e420f0eb","Type":"ContainerStarted","Data":"9a1783f49936d7ff4b9ef2768f3a2b0ff87283639c47621e4555630ce58a1fbb"}
Jul 14 14:56:36.336900 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:36.336852    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77" podStartSLOduration=1.336835766 podStartE2EDuration="1.336835766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:56:36.334911384 +0000 UTC m=+1574.001097804" watchObservedRunningTime="2026-07-14 14:56:36.336835766 +0000 UTC m=+1574.003022187"
Jul 14 14:56:37.314969 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:37.314934    2586 generic.go:356] "Generic (PLEG): container finished" podID="f8865865-2484-43fe-b15d-3f62e420f0eb" containerID="259b67cb2ea242bdbfa346eca40a1cb8c8d5b4f32e8bcfa3f7e4971db562d6cc" exitCode=7
Jul 14 14:56:37.315343 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:37.315021    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77" event={"ID":"f8865865-2484-43fe-b15d-3f62e420f0eb","Type":"ContainerDied","Data":"259b67cb2ea242bdbfa346eca40a1cb8c8d5b4f32e8bcfa3f7e4971db562d6cc"}
Jul 14 14:56:37.315343 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:37.315178    2586 scope.go:117] "RemoveContainer" containerID="259b67cb2ea242bdbfa346eca40a1cb8c8d5b4f32e8bcfa3f7e4971db562d6cc"
Jul 14 14:56:38.321407 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:38.321370    2586 generic.go:356] "Generic (PLEG): container finished" podID="f8865865-2484-43fe-b15d-3f62e420f0eb" containerID="2265aec03dea95f64bb3a20ec2fd0f6683a49adceaff9b6cea55e836ffbd6bd8" exitCode=7
Jul 14 14:56:38.321866 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:38.321467    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77" event={"ID":"f8865865-2484-43fe-b15d-3f62e420f0eb","Type":"ContainerDied","Data":"2265aec03dea95f64bb3a20ec2fd0f6683a49adceaff9b6cea55e836ffbd6bd8"}
Jul 14 14:56:38.321866 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:38.321582    2586 scope.go:117] "RemoveContainer" containerID="259b67cb2ea242bdbfa346eca40a1cb8c8d5b4f32e8bcfa3f7e4971db562d6cc"
Jul 14 14:56:38.321866 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:38.321807    2586 scope.go:117] "RemoveContainer" containerID="2265aec03dea95f64bb3a20ec2fd0f6683a49adceaff9b6cea55e836ffbd6bd8"
Jul 14 14:56:38.322031 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:56:38.322014    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-11424108-dcm77_odh-ai-gateway-infra(f8865865-2484-43fe-b15d-3f62e420f0eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77" podUID="f8865865-2484-43fe-b15d-3f62e420f0eb"
Jul 14 14:56:39.302296 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:39.302258    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:56:39.328963 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:39.328935    2586 scope.go:117] "RemoveContainer" containerID="2265aec03dea95f64bb3a20ec2fd0f6683a49adceaff9b6cea55e836ffbd6bd8"
Jul 14 14:56:39.329441 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:56:39.329180    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-11424108-dcm77_odh-ai-gateway-infra(f8865865-2484-43fe-b15d-3f62e420f0eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77" podUID="f8865865-2484-43fe-b15d-3f62e420f0eb"
Jul 14 14:56:53.871180 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:53.871143    2586 scope.go:117] "RemoveContainer" containerID="2265aec03dea95f64bb3a20ec2fd0f6683a49adceaff9b6cea55e836ffbd6bd8"
Jul 14 14:56:54.414049 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:54.414011    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77" event={"ID":"f8865865-2484-43fe-b15d-3f62e420f0eb","Type":"ContainerStarted","Data":"98675606ceb9c0e7c582af05559b45e723cbd189973347dbe52cd844b0065876"}
Jul 14 14:56:55.421537 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:55.421504    2586 generic.go:356] "Generic (PLEG): container finished" podID="f8865865-2484-43fe-b15d-3f62e420f0eb" containerID="98675606ceb9c0e7c582af05559b45e723cbd189973347dbe52cd844b0065876" exitCode=0
Jul 14 14:56:55.421987 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:55.421573    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77" event={"ID":"f8865865-2484-43fe-b15d-3f62e420f0eb","Type":"ContainerDied","Data":"98675606ceb9c0e7c582af05559b45e723cbd189973347dbe52cd844b0065876"}
Jul 14 14:56:55.421987 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:55.421636    2586 scope.go:117] "RemoveContainer" containerID="2265aec03dea95f64bb3a20ec2fd0f6683a49adceaff9b6cea55e836ffbd6bd8"
Jul 14 14:56:55.448680 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:55.448646    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77"]
Jul 14 14:56:56.557492 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:56.557469    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77"
Jul 14 14:56:56.602718 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:56.602685    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8865865-2484-43fe-b15d-3f62e420f0eb-maas-api-service-ca\") pod \"f8865865-2484-43fe-b15d-3f62e420f0eb\" (UID: \"f8865865-2484-43fe-b15d-3f62e420f0eb\") "
Jul 14 14:56:56.603135 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:56.603109    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8865865-2484-43fe-b15d-3f62e420f0eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8865865-2484-43fe-b15d-3f62e420f0eb" (UID: "f8865865-2484-43fe-b15d-3f62e420f0eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:56:56.703782 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:56.703693    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8865865-2484-43fe-b15d-3f62e420f0eb-maas-api-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:56:57.433500 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:57.433467    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77"
Jul 14 14:56:57.433500 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:57.433498    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77" event={"ID":"f8865865-2484-43fe-b15d-3f62e420f0eb","Type":"ContainerDied","Data":"9a1783f49936d7ff4b9ef2768f3a2b0ff87283639c47621e4555630ce58a1fbb"}
Jul 14 14:56:57.433747 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:57.433540    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a1783f49936d7ff4b9ef2768f3a2b0ff87283639c47621e4555630ce58a1fbb"
Jul 14 14:56:57.468983 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:57.468948    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77"]
Jul 14 14:56:57.472564 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:57.472533    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-dcm77"]
Jul 14 14:56:58.876123 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:56:58.876088    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8865865-2484-43fe-b15d-3f62e420f0eb" path="/var/lib/kubelet/pods/f8865865-2484-43fe-b15d-3f62e420f0eb/volumes"
Jul 14 14:57:35.403100 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.403010    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-sl42n"]
Jul 14 14:57:35.403549 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.403428    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8865865-2484-43fe-b15d-3f62e420f0eb" containerName="revoke-keys"
Jul 14 14:57:35.403549 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.403441    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8865865-2484-43fe-b15d-3f62e420f0eb" containerName="revoke-keys"
Jul 14 14:57:35.403549 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.403458    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8865865-2484-43fe-b15d-3f62e420f0eb" containerName="revoke-keys"
Jul 14 14:57:35.403549 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.403464    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8865865-2484-43fe-b15d-3f62e420f0eb" containerName="revoke-keys"
Jul 14 14:57:35.403549 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.403480    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8865865-2484-43fe-b15d-3f62e420f0eb" containerName="revoke-keys"
Jul 14 14:57:35.403549 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.403486    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8865865-2484-43fe-b15d-3f62e420f0eb" containerName="revoke-keys"
Jul 14 14:57:35.403549 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.403545    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8865865-2484-43fe-b15d-3f62e420f0eb" containerName="revoke-keys"
Jul 14 14:57:35.403549 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.403552    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8865865-2484-43fe-b15d-3f62e420f0eb" containerName="revoke-keys"
Jul 14 14:57:35.406968 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.406951    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-sl42n"
Jul 14 14:57:35.412031 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.412005    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5695q\""
Jul 14 14:57:35.414220 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.414191    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-sl42n"]
Jul 14 14:57:35.466484 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.466450    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c0b8bd-3911-482f-a28a-4b1ae46d7833-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-11424108-sl42n\" (UID: \"66c0b8bd-3911-482f-a28a-4b1ae46d7833\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-sl42n"
Jul 14 14:57:35.567570 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.567530    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c0b8bd-3911-482f-a28a-4b1ae46d7833-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-11424108-sl42n\" (UID: \"66c0b8bd-3911-482f-a28a-4b1ae46d7833\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-sl42n"
Jul 14 14:57:35.568029 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.568009    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c0b8bd-3911-482f-a28a-4b1ae46d7833-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-11424108-sl42n\" (UID: \"66c0b8bd-3911-482f-a28a-4b1ae46d7833\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-sl42n"
Jul 14 14:57:35.720160 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.720122    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-sl42n"
Jul 14 14:57:35.860387 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:35.860362    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-sl42n"]
Jul 14 14:57:35.862364 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:57:35.862340    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c0b8bd_3911_482f_a28a_4b1ae46d7833.slice/crio-bea1ac0cc710f785dca28be3d9a0685f6465b838411fbc6a56acf390a7a577bb WatchSource:0}: Error finding container bea1ac0cc710f785dca28be3d9a0685f6465b838411fbc6a56acf390a7a577bb: Status 404 returned error can't find the container with id bea1ac0cc710f785dca28be3d9a0685f6465b838411fbc6a56acf390a7a577bb
Jul 14 14:57:36.633656 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:36.633617    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-sl42n" event={"ID":"66c0b8bd-3911-482f-a28a-4b1ae46d7833","Type":"ContainerStarted","Data":"e53e245d88967b1c73048f09163d81d3ecead9843f7d3f8cee8280f241651d22"}
Jul 14 14:57:36.634104 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:36.633670    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-sl42n" event={"ID":"66c0b8bd-3911-482f-a28a-4b1ae46d7833","Type":"ContainerStarted","Data":"bea1ac0cc710f785dca28be3d9a0685f6465b838411fbc6a56acf390a7a577bb"}
Jul 14 14:57:37.640319 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:37.640280    2586 generic.go:356] "Generic (PLEG): container finished" podID="66c0b8bd-3911-482f-a28a-4b1ae46d7833" containerID="e53e245d88967b1c73048f09163d81d3ecead9843f7d3f8cee8280f241651d22" exitCode=0
Jul 14 14:57:37.640712 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:37.640357    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-sl42n" event={"ID":"66c0b8bd-3911-482f-a28a-4b1ae46d7833","Type":"ContainerDied","Data":"e53e245d88967b1c73048f09163d81d3ecead9843f7d3f8cee8280f241651d22"}
Jul 14 14:57:38.788739 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:38.788709    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-sl42n"
Jul 14 14:57:38.898253 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:38.898173    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c0b8bd-3911-482f-a28a-4b1ae46d7833-maas-api-service-ca\") pod \"66c0b8bd-3911-482f-a28a-4b1ae46d7833\" (UID: \"66c0b8bd-3911-482f-a28a-4b1ae46d7833\") "
Jul 14 14:57:38.898535 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:38.898515    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c0b8bd-3911-482f-a28a-4b1ae46d7833-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "66c0b8bd-3911-482f-a28a-4b1ae46d7833" (UID: "66c0b8bd-3911-482f-a28a-4b1ae46d7833"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:57:38.999383 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:38.999347    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c0b8bd-3911-482f-a28a-4b1ae46d7833-maas-api-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:57:39.651214 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:39.651175    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-sl42n" event={"ID":"66c0b8bd-3911-482f-a28a-4b1ae46d7833","Type":"ContainerDied","Data":"bea1ac0cc710f785dca28be3d9a0685f6465b838411fbc6a56acf390a7a577bb"}
Jul 14 14:57:39.651384 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:39.651225    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bea1ac0cc710f785dca28be3d9a0685f6465b838411fbc6a56acf390a7a577bb"
Jul 14 14:57:39.651384 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:39.651190    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-11424108-sl42n"
Jul 14 14:57:41.604192 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:41.604157    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"]
Jul 14 14:57:41.604954 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:41.604918    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk" podUID="aee2286c-8dd0-4444-b75e-56fedb771c46" containerName="maas-api" containerID="cri-o://0a98e1a3fd280d88b17a3ac82a579551a73799f25685b71532f5ac828a5039a8" gracePeriod=30
Jul 14 14:57:41.881079 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:41.881058    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:57:41.924993 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:41.924960    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee2286c-8dd0-4444-b75e-56fedb771c46-openshift-service-ca\") pod \"aee2286c-8dd0-4444-b75e-56fedb771c46\" (UID: \"aee2286c-8dd0-4444-b75e-56fedb771c46\") "
Jul 14 14:57:41.925168 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:41.925038    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee2286c-8dd0-4444-b75e-56fedb771c46-maas-api-tls\") pod \"aee2286c-8dd0-4444-b75e-56fedb771c46\" (UID: \"aee2286c-8dd0-4444-b75e-56fedb771c46\") "
Jul 14 14:57:41.925168 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:41.925060    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24b96\" (UniqueName: \"kubernetes.io/projected/aee2286c-8dd0-4444-b75e-56fedb771c46-kube-api-access-24b96\") pod \"aee2286c-8dd0-4444-b75e-56fedb771c46\" (UID: \"aee2286c-8dd0-4444-b75e-56fedb771c46\") "
Jul 14 14:57:41.925379 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:41.925350    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee2286c-8dd0-4444-b75e-56fedb771c46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aee2286c-8dd0-4444-b75e-56fedb771c46" (UID: "aee2286c-8dd0-4444-b75e-56fedb771c46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:57:41.927341 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:41.927310    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee2286c-8dd0-4444-b75e-56fedb771c46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aee2286c-8dd0-4444-b75e-56fedb771c46" (UID: "aee2286c-8dd0-4444-b75e-56fedb771c46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:57:41.927459 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:41.927387    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee2286c-8dd0-4444-b75e-56fedb771c46-kube-api-access-24b96" (OuterVolumeSpecName: "kube-api-access-24b96") pod "aee2286c-8dd0-4444-b75e-56fedb771c46" (UID: "aee2286c-8dd0-4444-b75e-56fedb771c46"). InnerVolumeSpecName "kube-api-access-24b96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:57:42.026015 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:42.025973    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee2286c-8dd0-4444-b75e-56fedb771c46-maas-api-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:57:42.026015 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:42.026008    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24b96\" (UniqueName: \"kubernetes.io/projected/aee2286c-8dd0-4444-b75e-56fedb771c46-kube-api-access-24b96\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:57:42.026015 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:42.026018    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee2286c-8dd0-4444-b75e-56fedb771c46-openshift-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:57:42.668134 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:42.668102    2586 generic.go:356] "Generic (PLEG): container finished" podID="aee2286c-8dd0-4444-b75e-56fedb771c46" containerID="0a98e1a3fd280d88b17a3ac82a579551a73799f25685b71532f5ac828a5039a8" exitCode=0
Jul 14 14:57:42.668542 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:42.668150    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"
Jul 14 14:57:42.668542 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:42.668198    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk" event={"ID":"aee2286c-8dd0-4444-b75e-56fedb771c46","Type":"ContainerDied","Data":"0a98e1a3fd280d88b17a3ac82a579551a73799f25685b71532f5ac828a5039a8"}
Jul 14 14:57:42.668542 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:42.668254    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk" event={"ID":"aee2286c-8dd0-4444-b75e-56fedb771c46","Type":"ContainerDied","Data":"02b21e2049c815aae45d40ba95237ac65867af8c375475d489a0ef2998fd2723"}
Jul 14 14:57:42.668542 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:42.668279    2586 scope.go:117] "RemoveContainer" containerID="0a98e1a3fd280d88b17a3ac82a579551a73799f25685b71532f5ac828a5039a8"
Jul 14 14:57:42.677769 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:42.677753    2586 scope.go:117] "RemoveContainer" containerID="0a98e1a3fd280d88b17a3ac82a579551a73799f25685b71532f5ac828a5039a8"
Jul 14 14:57:42.678037 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:57:42.678013    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a98e1a3fd280d88b17a3ac82a579551a73799f25685b71532f5ac828a5039a8\": container with ID starting with 0a98e1a3fd280d88b17a3ac82a579551a73799f25685b71532f5ac828a5039a8 not found: ID does not exist" containerID="0a98e1a3fd280d88b17a3ac82a579551a73799f25685b71532f5ac828a5039a8"
Jul 14 14:57:42.678104 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:42.678052    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a98e1a3fd280d88b17a3ac82a579551a73799f25685b71532f5ac828a5039a8"} err="failed to get container status \"0a98e1a3fd280d88b17a3ac82a579551a73799f25685b71532f5ac828a5039a8\": rpc error: code = NotFound desc = could not find container \"0a98e1a3fd280d88b17a3ac82a579551a73799f25685b71532f5ac828a5039a8\": container with ID starting with 0a98e1a3fd280d88b17a3ac82a579551a73799f25685b71532f5ac828a5039a8 not found: ID does not exist"
Jul 14 14:57:42.693031 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:42.693001    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"]
Jul 14 14:57:42.698721 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:42.698694    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-11424108-85d56c89d7-xfkfk"]
Jul 14 14:57:42.876956 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:42.876924    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aee2286c-8dd0-4444-b75e-56fedb771c46" path="/var/lib/kubelet/pods/aee2286c-8dd0-4444-b75e-56fedb771c46/volumes"
Jul 14 14:57:48.943681 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:48.943646    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"]
Jul 14 14:57:48.944231 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:48.944163    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c0b8bd-3911-482f-a28a-4b1ae46d7833" containerName="revoke-keys"
Jul 14 14:57:48.944231 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:48.944185    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c0b8bd-3911-482f-a28a-4b1ae46d7833" containerName="revoke-keys"
Jul 14 14:57:48.944231 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:48.944198    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee2286c-8dd0-4444-b75e-56fedb771c46" containerName="maas-api"
Jul 14 14:57:48.944231 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:48.944203    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee2286c-8dd0-4444-b75e-56fedb771c46" containerName="maas-api"
Jul 14 14:57:48.944468 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:48.944369    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee2286c-8dd0-4444-b75e-56fedb771c46" containerName="maas-api"
Jul 14 14:57:48.944468 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:48.944385    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c0b8bd-3911-482f-a28a-4b1ae46d7833" containerName="revoke-keys"
Jul 14 14:57:48.944468 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:48.944400    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8865865-2484-43fe-b15d-3f62e420f0eb" containerName="revoke-keys"
Jul 14 14:57:48.948943 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:48.948923    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:57:48.954920 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:48.954900    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9cfd4b9b-serving-cert\""
Jul 14 14:57:48.961159 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:48.961134    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"]
Jul 14 14:57:49.089748 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:49.089710    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr4hx\" (UniqueName: \"kubernetes.io/projected/05f8f9ce-4f2d-4373-acd4-7958eba08e95-kube-api-access-xr4hx\") pod \"maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k\" (UID: \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:57:49.089931 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:49.089760    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f8f9ce-4f2d-4373-acd4-7958eba08e95-maas-api-tls\") pod \"maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k\" (UID: \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:57:49.089931 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:49.089862    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f8f9ce-4f2d-4373-acd4-7958eba08e95-openshift-service-ca\") pod \"maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k\" (UID: \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:57:49.191163 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:49.191120    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr4hx\" (UniqueName: \"kubernetes.io/projected/05f8f9ce-4f2d-4373-acd4-7958eba08e95-kube-api-access-xr4hx\") pod \"maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k\" (UID: \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:57:49.191361 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:49.191173    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f8f9ce-4f2d-4373-acd4-7958eba08e95-maas-api-tls\") pod \"maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k\" (UID: \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:57:49.191361 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:49.191234    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f8f9ce-4f2d-4373-acd4-7958eba08e95-openshift-service-ca\") pod \"maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k\" (UID: \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:57:49.191541 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:57:49.191367    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-serving-cert: secret "maas-api-e2e-ait-9cfd4b9b-serving-cert" not found
Jul 14 14:57:49.191541 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:57:49.191488    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05f8f9ce-4f2d-4373-acd4-7958eba08e95-maas-api-tls podName:05f8f9ce-4f2d-4373-acd4-7958eba08e95 nodeName:}" failed. No retries permitted until 2026-07-14 14:57:49.69146307 +0000 UTC m=+1647.357649483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05f8f9ce-4f2d-4373-acd4-7958eba08e95-maas-api-tls") pod "maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k" (UID: "05f8f9ce-4f2d-4373-acd4-7958eba08e95") : secret "maas-api-e2e-ait-9cfd4b9b-serving-cert" not found
Jul 14 14:57:49.191915 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:49.191897    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f8f9ce-4f2d-4373-acd4-7958eba08e95-openshift-service-ca\") pod \"maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k\" (UID: \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:57:49.202992 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:49.202927    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr4hx\" (UniqueName: \"kubernetes.io/projected/05f8f9ce-4f2d-4373-acd4-7958eba08e95-kube-api-access-xr4hx\") pod \"maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k\" (UID: \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:57:49.695727 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:49.695692    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f8f9ce-4f2d-4373-acd4-7958eba08e95-maas-api-tls\") pod \"maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k\" (UID: \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:57:49.698261 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:49.698239    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f8f9ce-4f2d-4373-acd4-7958eba08e95-maas-api-tls\") pod \"maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k\" (UID: \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:57:49.859389 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:49.859353    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:57:50.016030 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:50.016002    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"]
Jul 14 14:57:50.016491 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:57:50.016463    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f8f9ce_4f2d_4373_acd4_7958eba08e95.slice/crio-052c89858d233974946b4a916e12e6975e6f4efa9fd661aa37f7b9416a0b3617 WatchSource:0}: Error finding container 052c89858d233974946b4a916e12e6975e6f4efa9fd661aa37f7b9416a0b3617: Status 404 returned error can't find the container with id 052c89858d233974946b4a916e12e6975e6f4efa9fd661aa37f7b9416a0b3617
Jul 14 14:57:50.711156 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:50.711122    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k" event={"ID":"05f8f9ce-4f2d-4373-acd4-7958eba08e95","Type":"ContainerStarted","Data":"052c89858d233974946b4a916e12e6975e6f4efa9fd661aa37f7b9416a0b3617"}
Jul 14 14:57:51.717317 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:51.717280    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k" event={"ID":"05f8f9ce-4f2d-4373-acd4-7958eba08e95","Type":"ContainerStarted","Data":"96145487cb44ef6b9f64891e12d3a0d5f93fd41c8192baef8ceb2e7a81d3c759"}
Jul 14 14:57:51.717749 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:51.717372    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:57:51.737862 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:51.737811    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k" podStartSLOduration=3.074080824 podStartE2EDuration="3.73779745s" totalImagesPullingTime="663.716626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:57:51.736058233 +0000 UTC m=+1649.402244653" watchObservedRunningTime="2026-07-14 14:57:51.73779745 +0000 UTC m=+1649.403983870"
Jul 14 14:57:53.804068 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:53.804034    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx"]
Jul 14 14:57:53.807540 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:53.807518    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx"
Jul 14 14:57:53.810318 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:53.810292    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5695q\""
Jul 14 14:57:53.814761 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:53.814726    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx"]
Jul 14 14:57:53.936698 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:53.936666    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c82f470-d1f3-4781-b3bc-d51f822df9a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx\" (UID: \"5c82f470-d1f3-4781-b3bc-d51f822df9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx"
Jul 14 14:57:54.037685 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:54.037643    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c82f470-d1f3-4781-b3bc-d51f822df9a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx\" (UID: \"5c82f470-d1f3-4781-b3bc-d51f822df9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx"
Jul 14 14:57:54.038273 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:54.038253    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c82f470-d1f3-4781-b3bc-d51f822df9a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx\" (UID: \"5c82f470-d1f3-4781-b3bc-d51f822df9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx"
Jul 14 14:57:54.119398 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:54.119299    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx"
Jul 14 14:57:54.259756 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:54.259589    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx"]
Jul 14 14:57:54.262527 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:57:54.262497    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c82f470_d1f3_4781_b3bc_d51f822df9a2.slice/crio-290b5c937fe705a7fde08ac959bfc014957ae29aa5c43a29fc188e63c7a73fd9 WatchSource:0}: Error finding container 290b5c937fe705a7fde08ac959bfc014957ae29aa5c43a29fc188e63c7a73fd9: Status 404 returned error can't find the container with id 290b5c937fe705a7fde08ac959bfc014957ae29aa5c43a29fc188e63c7a73fd9
Jul 14 14:57:54.734557 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:54.734525    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx" event={"ID":"5c82f470-d1f3-4781-b3bc-d51f822df9a2","Type":"ContainerStarted","Data":"77941ecdfd4b06a90421f3c746d8300e046d07f480c521eeb9c6d017a161ca21"}
Jul 14 14:57:54.734725 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:54.734578    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx" event={"ID":"5c82f470-d1f3-4781-b3bc-d51f822df9a2","Type":"ContainerStarted","Data":"290b5c937fe705a7fde08ac959bfc014957ae29aa5c43a29fc188e63c7a73fd9"}
Jul 14 14:57:54.751202 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:54.751074    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx" podStartSLOduration=1.7510537940000002 podStartE2EDuration="1.751053794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:57:54.750704712 +0000 UTC m=+1652.416891136" watchObservedRunningTime="2026-07-14 14:57:54.751053794 +0000 UTC m=+1652.417240215"
Jul 14 14:57:55.741159 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:55.741126    2586 generic.go:356] "Generic (PLEG): container finished" podID="5c82f470-d1f3-4781-b3bc-d51f822df9a2" containerID="77941ecdfd4b06a90421f3c746d8300e046d07f480c521eeb9c6d017a161ca21" exitCode=7
Jul 14 14:57:55.741587 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:55.741179    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx" event={"ID":"5c82f470-d1f3-4781-b3bc-d51f822df9a2","Type":"ContainerDied","Data":"77941ecdfd4b06a90421f3c746d8300e046d07f480c521eeb9c6d017a161ca21"}
Jul 14 14:57:55.741587 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:55.741478    2586 scope.go:117] "RemoveContainer" containerID="77941ecdfd4b06a90421f3c746d8300e046d07f480c521eeb9c6d017a161ca21"
Jul 14 14:57:56.747717 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:56.747681    2586 generic.go:356] "Generic (PLEG): container finished" podID="5c82f470-d1f3-4781-b3bc-d51f822df9a2" containerID="7d2ff24d9c6ccc74ea44c7cd694cf9d1221a78d453f7d903997aab567ab0bd07" exitCode=7
Jul 14 14:57:56.748114 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:56.747760    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx" event={"ID":"5c82f470-d1f3-4781-b3bc-d51f822df9a2","Type":"ContainerDied","Data":"7d2ff24d9c6ccc74ea44c7cd694cf9d1221a78d453f7d903997aab567ab0bd07"}
Jul 14 14:57:56.748114 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:56.747820    2586 scope.go:117] "RemoveContainer" containerID="77941ecdfd4b06a90421f3c746d8300e046d07f480c521eeb9c6d017a161ca21"
Jul 14 14:57:56.748218 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:56.748117    2586 scope.go:117] "RemoveContainer" containerID="7d2ff24d9c6ccc74ea44c7cd694cf9d1221a78d453f7d903997aab567ab0bd07"
Jul 14 14:57:56.748331 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:57:56.748312    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx_odh-ai-gateway-infra(5c82f470-d1f3-4781-b3bc-d51f822df9a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx" podUID="5c82f470-d1f3-4781-b3bc-d51f822df9a2"
Jul 14 14:57:57.727757 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:57.727727    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:57:57.754513 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:57:57.754488    2586 scope.go:117] "RemoveContainer" containerID="7d2ff24d9c6ccc74ea44c7cd694cf9d1221a78d453f7d903997aab567ab0bd07"
Jul 14 14:57:57.754887 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:57:57.754744    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx_odh-ai-gateway-infra(5c82f470-d1f3-4781-b3bc-d51f822df9a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx" podUID="5c82f470-d1f3-4781-b3bc-d51f822df9a2"
Jul 14 14:58:09.870768 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:09.870734    2586 scope.go:117] "RemoveContainer" containerID="7d2ff24d9c6ccc74ea44c7cd694cf9d1221a78d453f7d903997aab567ab0bd07"
Jul 14 14:58:10.821759 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:10.821725    2586 generic.go:356] "Generic (PLEG): container finished" podID="5c82f470-d1f3-4781-b3bc-d51f822df9a2" containerID="234962364fbc1822ff45a6f9cd45526cbc2c9548f82db3f26e58109592564885" exitCode=0
Jul 14 14:58:10.821924 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:10.821780    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx" event={"ID":"5c82f470-d1f3-4781-b3bc-d51f822df9a2","Type":"ContainerDied","Data":"234962364fbc1822ff45a6f9cd45526cbc2c9548f82db3f26e58109592564885"}
Jul 14 14:58:10.821924 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:10.821827    2586 scope.go:117] "RemoveContainer" containerID="7d2ff24d9c6ccc74ea44c7cd694cf9d1221a78d453f7d903997aab567ab0bd07"
Jul 14 14:58:11.861794 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:11.861755    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx"]
Jul 14 14:58:11.968327 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:11.968305    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx"
Jul 14 14:58:12.006055 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:12.006003    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c82f470-d1f3-4781-b3bc-d51f822df9a2-maas-api-service-ca\") pod \"5c82f470-d1f3-4781-b3bc-d51f822df9a2\" (UID: \"5c82f470-d1f3-4781-b3bc-d51f822df9a2\") "
Jul 14 14:58:12.006486 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:12.006456    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c82f470-d1f3-4781-b3bc-d51f822df9a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c82f470-d1f3-4781-b3bc-d51f822df9a2" (UID: "5c82f470-d1f3-4781-b3bc-d51f822df9a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:58:12.107964 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:12.107863    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c82f470-d1f3-4781-b3bc-d51f822df9a2-maas-api-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:58:12.834888 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:12.834860    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx"
Jul 14 14:58:12.834888 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:12.834867    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx" event={"ID":"5c82f470-d1f3-4781-b3bc-d51f822df9a2","Type":"ContainerDied","Data":"290b5c937fe705a7fde08ac959bfc014957ae29aa5c43a29fc188e63c7a73fd9"}
Jul 14 14:58:12.835105 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:12.834937    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="290b5c937fe705a7fde08ac959bfc014957ae29aa5c43a29fc188e63c7a73fd9"
Jul 14 14:58:12.878147 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:12.878116    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx"]
Jul 14 14:58:12.878529 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:12.878195    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-97vvx"]
Jul 14 14:58:14.875939 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:14.875896    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c82f470-d1f3-4781-b3bc-d51f822df9a2" path="/var/lib/kubelet/pods/5c82f470-d1f3-4781-b3bc-d51f822df9a2/volumes"
Jul 14 14:58:43.852964 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:43.852875    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j"]
Jul 14 14:58:43.853536 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:43.853516    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c82f470-d1f3-4781-b3bc-d51f822df9a2" containerName="revoke-keys"
Jul 14 14:58:43.853589 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:43.853565    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c82f470-d1f3-4781-b3bc-d51f822df9a2" containerName="revoke-keys"
Jul 14 14:58:43.853589 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:43.853580    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c82f470-d1f3-4781-b3bc-d51f822df9a2" containerName="revoke-keys"
Jul 14 14:58:43.853660 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:43.853589    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c82f470-d1f3-4781-b3bc-d51f822df9a2" containerName="revoke-keys"
Jul 14 14:58:43.853731 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:43.853719    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c82f470-d1f3-4781-b3bc-d51f822df9a2" containerName="revoke-keys"
Jul 14 14:58:43.853781 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:43.853743    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c82f470-d1f3-4781-b3bc-d51f822df9a2" containerName="revoke-keys"
Jul 14 14:58:43.858580 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:43.858560    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j"
Jul 14 14:58:43.861493 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:43.861469    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5695q\""
Jul 14 14:58:43.865805 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:43.865773    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j"]
Jul 14 14:58:44.007981 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:44.007945    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e734f3-56eb-4ebb-91b7-851263fc0e33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j\" (UID: \"25e734f3-56eb-4ebb-91b7-851263fc0e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j"
Jul 14 14:58:44.109153 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:44.109061    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e734f3-56eb-4ebb-91b7-851263fc0e33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j\" (UID: \"25e734f3-56eb-4ebb-91b7-851263fc0e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j"
Jul 14 14:58:44.109711 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:44.109691    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e734f3-56eb-4ebb-91b7-851263fc0e33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j\" (UID: \"25e734f3-56eb-4ebb-91b7-851263fc0e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j"
Jul 14 14:58:44.170778 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:44.170745    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j"
Jul 14 14:58:44.310944 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:44.310912    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j"]
Jul 14 14:58:44.312734 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:58:44.312709    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e734f3_56eb_4ebb_91b7_851263fc0e33.slice/crio-52580172f3161fa928eb69c7943eb064d2090699a312c5bd978868bdb8b79819 WatchSource:0}: Error finding container 52580172f3161fa928eb69c7943eb064d2090699a312c5bd978868bdb8b79819: Status 404 returned error can't find the container with id 52580172f3161fa928eb69c7943eb064d2090699a312c5bd978868bdb8b79819
Jul 14 14:58:44.995807 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:44.995763    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j" event={"ID":"25e734f3-56eb-4ebb-91b7-851263fc0e33","Type":"ContainerStarted","Data":"b931255908d695356bd6e3c901b5840f3c33fc3fb98fbd5feb762fb610d10155"}
Jul 14 14:58:44.996215 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:44.995817    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j" event={"ID":"25e734f3-56eb-4ebb-91b7-851263fc0e33","Type":"ContainerStarted","Data":"52580172f3161fa928eb69c7943eb064d2090699a312c5bd978868bdb8b79819"}
Jul 14 14:58:45.013867 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:45.013819    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j" podStartSLOduration=2.013804424 podStartE2EDuration="2.013804424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:58:45.011260048 +0000 UTC m=+1702.677446468" watchObservedRunningTime="2026-07-14 14:58:45.013804424 +0000 UTC m=+1702.679990845"
Jul 14 14:58:46.002601 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:46.002565    2586 generic.go:356] "Generic (PLEG): container finished" podID="25e734f3-56eb-4ebb-91b7-851263fc0e33" containerID="b931255908d695356bd6e3c901b5840f3c33fc3fb98fbd5feb762fb610d10155" exitCode=0
Jul 14 14:58:46.002957 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:46.002648    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j" event={"ID":"25e734f3-56eb-4ebb-91b7-851263fc0e33","Type":"ContainerDied","Data":"b931255908d695356bd6e3c901b5840f3c33fc3fb98fbd5feb762fb610d10155"}
Jul 14 14:58:47.136778 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:47.136718    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j"
Jul 14 14:58:47.137810 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:47.137783    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e734f3-56eb-4ebb-91b7-851263fc0e33-maas-api-service-ca\") pod \"25e734f3-56eb-4ebb-91b7-851263fc0e33\" (UID: \"25e734f3-56eb-4ebb-91b7-851263fc0e33\") "
Jul 14 14:58:47.138124 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:47.138101    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e734f3-56eb-4ebb-91b7-851263fc0e33-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25e734f3-56eb-4ebb-91b7-851263fc0e33" (UID: "25e734f3-56eb-4ebb-91b7-851263fc0e33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:58:47.239324 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:47.239293    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e734f3-56eb-4ebb-91b7-851263fc0e33-maas-api-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:58:48.014199 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:48.014172    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j"
Jul 14 14:58:48.014368 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:48.014222    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j" event={"ID":"25e734f3-56eb-4ebb-91b7-851263fc0e33","Type":"ContainerDied","Data":"52580172f3161fa928eb69c7943eb064d2090699a312c5bd978868bdb8b79819"}
Jul 14 14:58:48.014368 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:48.014269    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52580172f3161fa928eb69c7943eb064d2090699a312c5bd978868bdb8b79819"
Jul 14 14:58:54.315129 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:54.313226    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"]
Jul 14 14:58:54.315129 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:54.313503    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k" podUID="05f8f9ce-4f2d-4373-acd4-7958eba08e95" containerName="maas-api" containerID="cri-o://96145487cb44ef6b9f64891e12d3a0d5f93fd41c8192baef8ceb2e7a81d3c759" gracePeriod=30
Jul 14 14:58:54.559974 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:54.559950    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:58:54.606876 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:54.606796    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr4hx\" (UniqueName: \"kubernetes.io/projected/05f8f9ce-4f2d-4373-acd4-7958eba08e95-kube-api-access-xr4hx\") pod \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\" (UID: \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\") "
Jul 14 14:58:54.607016 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:54.606887    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f8f9ce-4f2d-4373-acd4-7958eba08e95-maas-api-tls\") pod \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\" (UID: \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\") "
Jul 14 14:58:54.607016 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:54.606910    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f8f9ce-4f2d-4373-acd4-7958eba08e95-openshift-service-ca\") pod \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\" (UID: \"05f8f9ce-4f2d-4373-acd4-7958eba08e95\") "
Jul 14 14:58:54.607342 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:54.607312    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f8f9ce-4f2d-4373-acd4-7958eba08e95-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05f8f9ce-4f2d-4373-acd4-7958eba08e95" (UID: "05f8f9ce-4f2d-4373-acd4-7958eba08e95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:58:54.609029 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:54.609005    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f8f9ce-4f2d-4373-acd4-7958eba08e95-kube-api-access-xr4hx" (OuterVolumeSpecName: "kube-api-access-xr4hx") pod "05f8f9ce-4f2d-4373-acd4-7958eba08e95" (UID: "05f8f9ce-4f2d-4373-acd4-7958eba08e95"). InnerVolumeSpecName "kube-api-access-xr4hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:58:54.609131 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:54.609104    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f8f9ce-4f2d-4373-acd4-7958eba08e95-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05f8f9ce-4f2d-4373-acd4-7958eba08e95" (UID: "05f8f9ce-4f2d-4373-acd4-7958eba08e95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:58:54.708117 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:54.708078    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f8f9ce-4f2d-4373-acd4-7958eba08e95-maas-api-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:58:54.708117 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:54.708110    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f8f9ce-4f2d-4373-acd4-7958eba08e95-openshift-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:58:54.708117 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:54.708122    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr4hx\" (UniqueName: \"kubernetes.io/projected/05f8f9ce-4f2d-4373-acd4-7958eba08e95-kube-api-access-xr4hx\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:58:55.051163 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:55.051131    2586 generic.go:356] "Generic (PLEG): container finished" podID="05f8f9ce-4f2d-4373-acd4-7958eba08e95" containerID="96145487cb44ef6b9f64891e12d3a0d5f93fd41c8192baef8ceb2e7a81d3c759" exitCode=0
Jul 14 14:58:55.051301 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:55.051175    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k" event={"ID":"05f8f9ce-4f2d-4373-acd4-7958eba08e95","Type":"ContainerDied","Data":"96145487cb44ef6b9f64891e12d3a0d5f93fd41c8192baef8ceb2e7a81d3c759"}
Jul 14 14:58:55.051301 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:55.051184    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"
Jul 14 14:58:55.051301 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:55.051217    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k" event={"ID":"05f8f9ce-4f2d-4373-acd4-7958eba08e95","Type":"ContainerDied","Data":"052c89858d233974946b4a916e12e6975e6f4efa9fd661aa37f7b9416a0b3617"}
Jul 14 14:58:55.051301 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:55.051239    2586 scope.go:117] "RemoveContainer" containerID="96145487cb44ef6b9f64891e12d3a0d5f93fd41c8192baef8ceb2e7a81d3c759"
Jul 14 14:58:55.060379 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:55.060361    2586 scope.go:117] "RemoveContainer" containerID="96145487cb44ef6b9f64891e12d3a0d5f93fd41c8192baef8ceb2e7a81d3c759"
Jul 14 14:58:55.060653 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:58:55.060635    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96145487cb44ef6b9f64891e12d3a0d5f93fd41c8192baef8ceb2e7a81d3c759\": container with ID starting with 96145487cb44ef6b9f64891e12d3a0d5f93fd41c8192baef8ceb2e7a81d3c759 not found: ID does not exist" containerID="96145487cb44ef6b9f64891e12d3a0d5f93fd41c8192baef8ceb2e7a81d3c759"
Jul 14 14:58:55.060724 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:55.060662    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96145487cb44ef6b9f64891e12d3a0d5f93fd41c8192baef8ceb2e7a81d3c759"} err="failed to get container status \"96145487cb44ef6b9f64891e12d3a0d5f93fd41c8192baef8ceb2e7a81d3c759\": rpc error: code = NotFound desc = could not find container \"96145487cb44ef6b9f64891e12d3a0d5f93fd41c8192baef8ceb2e7a81d3c759\": container with ID starting with 96145487cb44ef6b9f64891e12d3a0d5f93fd41c8192baef8ceb2e7a81d3c759 not found: ID does not exist"
Jul 14 14:58:55.073185 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:55.073110    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"]
Jul 14 14:58:55.077852 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:55.077824    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9cfd4b9b-7964d464f4-n7t4k"]
Jul 14 14:58:56.877465 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:58:56.877398    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05f8f9ce-4f2d-4373-acd4-7958eba08e95" path="/var/lib/kubelet/pods/05f8f9ce-4f2d-4373-acd4-7958eba08e95/volumes"
Jul 14 14:59:11.578071 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.578036    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"]
Jul 14 14:59:11.578633 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.578618    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e734f3-56eb-4ebb-91b7-851263fc0e33" containerName="revoke-keys"
Jul 14 14:59:11.578696 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.578637    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e734f3-56eb-4ebb-91b7-851263fc0e33" containerName="revoke-keys"
Jul 14 14:59:11.578696 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.578669    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c82f470-d1f3-4781-b3bc-d51f822df9a2" containerName="revoke-keys"
Jul 14 14:59:11.578696 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.578677    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c82f470-d1f3-4781-b3bc-d51f822df9a2" containerName="revoke-keys"
Jul 14 14:59:11.578696 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.578694    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f8f9ce-4f2d-4373-acd4-7958eba08e95" containerName="maas-api"
Jul 14 14:59:11.578853 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.578703    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f8f9ce-4f2d-4373-acd4-7958eba08e95" containerName="maas-api"
Jul 14 14:59:11.578853 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.578785    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f8f9ce-4f2d-4373-acd4-7958eba08e95" containerName="maas-api"
Jul 14 14:59:11.578853 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.578796    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c82f470-d1f3-4781-b3bc-d51f822df9a2" containerName="revoke-keys"
Jul 14 14:59:11.578853 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.578805    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="25e734f3-56eb-4ebb-91b7-851263fc0e33" containerName="revoke-keys"
Jul 14 14:59:11.583956 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.583922    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 14:59:11.587447 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.587396    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4b164646-serving-cert\""
Jul 14 14:59:11.596189 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.596164    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"]
Jul 14 14:59:11.671683 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.671651    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64d7k\" (UniqueName: \"kubernetes.io/projected/075f5255-1fc7-4016-b525-6d0bdafdca56-kube-api-access-64d7k\") pod \"maas-api-e2e-derive-4b164646-74bbfdd487-q97bp\" (UID: \"075f5255-1fc7-4016-b525-6d0bdafdca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 14:59:11.671842 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.671716    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075f5255-1fc7-4016-b525-6d0bdafdca56-maas-api-tls\") pod \"maas-api-e2e-derive-4b164646-74bbfdd487-q97bp\" (UID: \"075f5255-1fc7-4016-b525-6d0bdafdca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 14:59:11.671842 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.671757    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075f5255-1fc7-4016-b525-6d0bdafdca56-openshift-service-ca\") pod \"maas-api-e2e-derive-4b164646-74bbfdd487-q97bp\" (UID: \"075f5255-1fc7-4016-b525-6d0bdafdca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 14:59:11.772978 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.772941    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075f5255-1fc7-4016-b525-6d0bdafdca56-maas-api-tls\") pod \"maas-api-e2e-derive-4b164646-74bbfdd487-q97bp\" (UID: \"075f5255-1fc7-4016-b525-6d0bdafdca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 14:59:11.773156 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.773007    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075f5255-1fc7-4016-b525-6d0bdafdca56-openshift-service-ca\") pod \"maas-api-e2e-derive-4b164646-74bbfdd487-q97bp\" (UID: \"075f5255-1fc7-4016-b525-6d0bdafdca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 14:59:11.773156 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:59:11.773097    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-serving-cert: secret "maas-api-e2e-derive-4b164646-serving-cert" not found
Jul 14 14:59:11.773156 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.773104    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64d7k\" (UniqueName: \"kubernetes.io/projected/075f5255-1fc7-4016-b525-6d0bdafdca56-kube-api-access-64d7k\") pod \"maas-api-e2e-derive-4b164646-74bbfdd487-q97bp\" (UID: \"075f5255-1fc7-4016-b525-6d0bdafdca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 14:59:11.773260 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:59:11.773167    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/075f5255-1fc7-4016-b525-6d0bdafdca56-maas-api-tls podName:075f5255-1fc7-4016-b525-6d0bdafdca56 nodeName:}" failed. No retries permitted until 2026-07-14 14:59:12.273146502 +0000 UTC m=+1729.939332920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/075f5255-1fc7-4016-b525-6d0bdafdca56-maas-api-tls") pod "maas-api-e2e-derive-4b164646-74bbfdd487-q97bp" (UID: "075f5255-1fc7-4016-b525-6d0bdafdca56") : secret "maas-api-e2e-derive-4b164646-serving-cert" not found
Jul 14 14:59:11.773697 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.773679    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075f5255-1fc7-4016-b525-6d0bdafdca56-openshift-service-ca\") pod \"maas-api-e2e-derive-4b164646-74bbfdd487-q97bp\" (UID: \"075f5255-1fc7-4016-b525-6d0bdafdca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 14:59:11.783187 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:11.783156    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64d7k\" (UniqueName: \"kubernetes.io/projected/075f5255-1fc7-4016-b525-6d0bdafdca56-kube-api-access-64d7k\") pod \"maas-api-e2e-derive-4b164646-74bbfdd487-q97bp\" (UID: \"075f5255-1fc7-4016-b525-6d0bdafdca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 14:59:12.277795 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:12.277746    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075f5255-1fc7-4016-b525-6d0bdafdca56-maas-api-tls\") pod \"maas-api-e2e-derive-4b164646-74bbfdd487-q97bp\" (UID: \"075f5255-1fc7-4016-b525-6d0bdafdca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 14:59:12.280500 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:12.280476    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075f5255-1fc7-4016-b525-6d0bdafdca56-maas-api-tls\") pod \"maas-api-e2e-derive-4b164646-74bbfdd487-q97bp\" (UID: \"075f5255-1fc7-4016-b525-6d0bdafdca56\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 14:59:12.498515 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:12.498471    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 14:59:12.640158 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:12.640136    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"]
Jul 14 14:59:12.642078 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:59:12.642050    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075f5255_1fc7_4016_b525_6d0bdafdca56.slice/crio-e804d42511769fe708c31a670ed06f12e58b7028ba364228e7b651a6b4ecd078 WatchSource:0}: Error finding container e804d42511769fe708c31a670ed06f12e58b7028ba364228e7b651a6b4ecd078: Status 404 returned error can't find the container with id e804d42511769fe708c31a670ed06f12e58b7028ba364228e7b651a6b4ecd078
Jul 14 14:59:13.148428 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:13.148378    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp" event={"ID":"075f5255-1fc7-4016-b525-6d0bdafdca56","Type":"ContainerStarted","Data":"e804d42511769fe708c31a670ed06f12e58b7028ba364228e7b651a6b4ecd078"}
Jul 14 14:59:14.155529 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:14.155491    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp" event={"ID":"075f5255-1fc7-4016-b525-6d0bdafdca56","Type":"ContainerStarted","Data":"2da4dc4cb8328f85bae92b9c46c93de4c3dd23072b7b0089e0bfb781c6c4d5a1"}
Jul 14 14:59:14.155917 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:14.155597    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 14:59:14.182065 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:14.182011    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp" podStartSLOduration=2.469419125 podStartE2EDuration="3.181994352s" totalImagesPullingTime="712.575227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:59:14.179103492 +0000 UTC m=+1731.845289910" watchObservedRunningTime="2026-07-14 14:59:14.181994352 +0000 UTC m=+1731.848180773"
Jul 14 14:59:15.763569 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:15.763532    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9"]
Jul 14 14:59:15.767155 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:15.767129    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9"
Jul 14 14:59:15.770280 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:15.770236    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5695q\""
Jul 14 14:59:15.773509 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:15.773484    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9"]
Jul 14 14:59:15.912042 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:15.912008    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c598858-91fd-47c3-96cd-e34ed5b00018-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4b164646-wxhv9\" (UID: \"4c598858-91fd-47c3-96cd-e34ed5b00018\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9"
Jul 14 14:59:16.013432 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:16.013378    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c598858-91fd-47c3-96cd-e34ed5b00018-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4b164646-wxhv9\" (UID: \"4c598858-91fd-47c3-96cd-e34ed5b00018\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9"
Jul 14 14:59:16.014037 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:16.013991    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c598858-91fd-47c3-96cd-e34ed5b00018-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4b164646-wxhv9\" (UID: \"4c598858-91fd-47c3-96cd-e34ed5b00018\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9"
Jul 14 14:59:16.080488 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:16.080446    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9"
Jul 14 14:59:16.222024 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:16.221997    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9"]
Jul 14 14:59:16.223890 ip-10-0-141-209 kubenswrapper[2586]: W0714 14:59:16.223864    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c598858_91fd_47c3_96cd_e34ed5b00018.slice/crio-208d9a829983da291da51180eecae9b3b1e40a33be20df02f1df45435da9f51a WatchSource:0}: Error finding container 208d9a829983da291da51180eecae9b3b1e40a33be20df02f1df45435da9f51a: Status 404 returned error can't find the container with id 208d9a829983da291da51180eecae9b3b1e40a33be20df02f1df45435da9f51a
Jul 14 14:59:17.174923 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:17.174887    2586 generic.go:356] "Generic (PLEG): container finished" podID="4c598858-91fd-47c3-96cd-e34ed5b00018" containerID="b1cab04cac7fae4379b5d1f9fc7aa05665788601be258c2eed3205906b60f234" exitCode=7
Jul 14 14:59:17.175353 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:17.174965    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9" event={"ID":"4c598858-91fd-47c3-96cd-e34ed5b00018","Type":"ContainerDied","Data":"b1cab04cac7fae4379b5d1f9fc7aa05665788601be258c2eed3205906b60f234"}
Jul 14 14:59:17.175353 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:17.175017    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9" event={"ID":"4c598858-91fd-47c3-96cd-e34ed5b00018","Type":"ContainerStarted","Data":"208d9a829983da291da51180eecae9b3b1e40a33be20df02f1df45435da9f51a"}
Jul 14 14:59:17.175353 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:17.175164    2586 scope.go:117] "RemoveContainer" containerID="b1cab04cac7fae4379b5d1f9fc7aa05665788601be258c2eed3205906b60f234"
Jul 14 14:59:18.183550 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:18.183514    2586 generic.go:356] "Generic (PLEG): container finished" podID="4c598858-91fd-47c3-96cd-e34ed5b00018" containerID="d85380e2b7bdb442d64cc83384f57e39e4ef3a52bb5fd41118d5141af1f1f1e9" exitCode=7
Jul 14 14:59:18.184011 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:18.183573    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9" event={"ID":"4c598858-91fd-47c3-96cd-e34ed5b00018","Type":"ContainerDied","Data":"d85380e2b7bdb442d64cc83384f57e39e4ef3a52bb5fd41118d5141af1f1f1e9"}
Jul 14 14:59:18.184011 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:18.183623    2586 scope.go:117] "RemoveContainer" containerID="b1cab04cac7fae4379b5d1f9fc7aa05665788601be258c2eed3205906b60f234"
Jul 14 14:59:18.184011 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:18.183806    2586 scope.go:117] "RemoveContainer" containerID="d85380e2b7bdb442d64cc83384f57e39e4ef3a52bb5fd41118d5141af1f1f1e9"
Jul 14 14:59:18.184166 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:59:18.184059    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4b164646-wxhv9_odh-ai-gateway-infra(4c598858-91fd-47c3-96cd-e34ed5b00018)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9" podUID="4c598858-91fd-47c3-96cd-e34ed5b00018"
Jul 14 14:59:19.190329 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:19.190298    2586 scope.go:117] "RemoveContainer" containerID="d85380e2b7bdb442d64cc83384f57e39e4ef3a52bb5fd41118d5141af1f1f1e9"
Jul 14 14:59:19.190771 ip-10-0-141-209 kubenswrapper[2586]: E0714 14:59:19.190556    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4b164646-wxhv9_odh-ai-gateway-infra(4c598858-91fd-47c3-96cd-e34ed5b00018)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9" podUID="4c598858-91fd-47c3-96cd-e34ed5b00018"
Jul 14 14:59:20.166760 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:20.166732    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 14:59:30.870526 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:30.870489    2586 scope.go:117] "RemoveContainer" containerID="d85380e2b7bdb442d64cc83384f57e39e4ef3a52bb5fd41118d5141af1f1f1e9"
Jul 14 14:59:31.253514 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:31.253479    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9" event={"ID":"4c598858-91fd-47c3-96cd-e34ed5b00018","Type":"ContainerStarted","Data":"ba40f2ffed6dfc0098392722811d3e7f08f1cfb036c19070edb00de4f7a22709"}
Jul 14 14:59:31.270306 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:31.270253    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9" podStartSLOduration=16.270237497 podStartE2EDuration="16.270237497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:59:31.269696144 +0000 UTC m=+1748.935882565" watchObservedRunningTime="2026-07-14 14:59:31.270237497 +0000 UTC m=+1748.936423917"
Jul 14 14:59:32.260097 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:32.260063    2586 generic.go:356] "Generic (PLEG): container finished" podID="4c598858-91fd-47c3-96cd-e34ed5b00018" containerID="ba40f2ffed6dfc0098392722811d3e7f08f1cfb036c19070edb00de4f7a22709" exitCode=0
Jul 14 14:59:32.260515 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:32.260136    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9" event={"ID":"4c598858-91fd-47c3-96cd-e34ed5b00018","Type":"ContainerDied","Data":"ba40f2ffed6dfc0098392722811d3e7f08f1cfb036c19070edb00de4f7a22709"}
Jul 14 14:59:32.260515 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:32.260222    2586 scope.go:117] "RemoveContainer" containerID="d85380e2b7bdb442d64cc83384f57e39e4ef3a52bb5fd41118d5141af1f1f1e9"
Jul 14 14:59:32.288962 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:32.288928    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9"]
Jul 14 14:59:33.406181 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:33.406156    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9"
Jul 14 14:59:33.465602 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:33.465567    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c598858-91fd-47c3-96cd-e34ed5b00018-maas-api-service-ca\") pod \"4c598858-91fd-47c3-96cd-e34ed5b00018\" (UID: \"4c598858-91fd-47c3-96cd-e34ed5b00018\") "
Jul 14 14:59:33.465920 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:33.465898    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c598858-91fd-47c3-96cd-e34ed5b00018-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c598858-91fd-47c3-96cd-e34ed5b00018" (UID: "4c598858-91fd-47c3-96cd-e34ed5b00018"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:59:33.566556 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:33.566465    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c598858-91fd-47c3-96cd-e34ed5b00018-maas-api-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 14:59:34.273819 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:34.273787    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9"
Jul 14 14:59:34.274012 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:34.273830    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9" event={"ID":"4c598858-91fd-47c3-96cd-e34ed5b00018","Type":"ContainerDied","Data":"208d9a829983da291da51180eecae9b3b1e40a33be20df02f1df45435da9f51a"}
Jul 14 14:59:34.274012 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:34.273884    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="208d9a829983da291da51180eecae9b3b1e40a33be20df02f1df45435da9f51a"
Jul 14 14:59:34.307818 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:34.307784    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9"]
Jul 14 14:59:34.310725 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:34.310700    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-wxhv9"]
Jul 14 14:59:34.876769 ip-10-0-141-209 kubenswrapper[2586]: I0714 14:59:34.876737    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c598858-91fd-47c3-96cd-e34ed5b00018" path="/var/lib/kubelet/pods/4c598858-91fd-47c3-96cd-e34ed5b00018/volumes"
Jul 14 15:00:00.138577 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.138543    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734020-54d2m"]
Jul 14 15:00:00.140895 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.138919    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c598858-91fd-47c3-96cd-e34ed5b00018" containerName="revoke-keys"
Jul 14 15:00:00.140895 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.138930    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c598858-91fd-47c3-96cd-e34ed5b00018" containerName="revoke-keys"
Jul 14 15:00:00.140895 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.138946    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c598858-91fd-47c3-96cd-e34ed5b00018" containerName="revoke-keys"
Jul 14 15:00:00.140895 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.138951    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c598858-91fd-47c3-96cd-e34ed5b00018" containerName="revoke-keys"
Jul 14 15:00:00.140895 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.139017    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c598858-91fd-47c3-96cd-e34ed5b00018" containerName="revoke-keys"
Jul 14 15:00:00.140895 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.139025    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c598858-91fd-47c3-96cd-e34ed5b00018" containerName="revoke-keys"
Jul 14 15:00:00.141921 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.141905    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734020-54d2m"
Jul 14 15:00:00.144983 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.144964    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5695q\""
Jul 14 15:00:00.145730 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.145712    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4b164646-29734020-fb8sf"]
Jul 14 15:00:00.146059 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.146049    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c598858-91fd-47c3-96cd-e34ed5b00018" containerName="revoke-keys"
Jul 14 15:00:00.146101 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.146061    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c598858-91fd-47c3-96cd-e34ed5b00018" containerName="revoke-keys"
Jul 14 15:00:00.146144 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.146136    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c598858-91fd-47c3-96cd-e34ed5b00018" containerName="revoke-keys"
Jul 14 15:00:00.149061 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.149041    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4b164646-29734020-fb8sf"
Jul 14 15:00:00.150820 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.150795    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734020-54d2m"]
Jul 14 15:00:00.155684 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.155502    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734020-54d2m"
Jul 14 15:00:00.160899 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.159371    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4b164646-29734020-fb8sf"]
Jul 14 15:00:00.167309 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.167284    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4b164646-29734020-fb8sf"
Jul 14 15:00:00.327990 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.327715    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734020-54d2m"]
Jul 14 15:00:00.328890 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:00:00.328852    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971bdfa3_4000_431a_bc2a_a88cb66599ee.slice/crio-59edae321860b2fae22cdd0f955ac86def7ed3de501f0c2fbc2756bd2c623294 WatchSource:0}: Error finding container 59edae321860b2fae22cdd0f955ac86def7ed3de501f0c2fbc2756bd2c623294: Status 404 returned error can't find the container with id 59edae321860b2fae22cdd0f955ac86def7ed3de501f0c2fbc2756bd2c623294
Jul 14 15:00:00.356509 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.356479    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4b164646-29734020-fb8sf"]
Jul 14 15:00:00.357800 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:00:00.357776    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79d1d5f_5dab_49a0_bb04_ed32cd858c52.slice/crio-dc8564933ac7d53c4ae2406c879bdd74bb52c87231c78b91bf40f59874ef5810 WatchSource:0}: Error finding container dc8564933ac7d53c4ae2406c879bdd74bb52c87231c78b91bf40f59874ef5810: Status 404 returned error can't find the container with id dc8564933ac7d53c4ae2406c879bdd74bb52c87231c78b91bf40f59874ef5810
Jul 14 15:00:00.405206 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.405119    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4b164646-29734020-fb8sf" event={"ID":"a79d1d5f-5dab-49a0-bb04-ed32cd858c52","Type":"ContainerStarted","Data":"dc8564933ac7d53c4ae2406c879bdd74bb52c87231c78b91bf40f59874ef5810"}
Jul 14 15:00:00.406375 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:00.406342    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734020-54d2m" event={"ID":"971bdfa3-4000-431a-bc2a-a88cb66599ee","Type":"ContainerStarted","Data":"59edae321860b2fae22cdd0f955ac86def7ed3de501f0c2fbc2756bd2c623294"}
Jul 14 15:00:01.413134 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:01.413099    2586 generic.go:356] "Generic (PLEG): container finished" podID="a79d1d5f-5dab-49a0-bb04-ed32cd858c52" containerID="094caee4350742d170bcd6eb39db54e3b4859029daf043c457143f1297798832" exitCode=0
Jul 14 15:00:01.413581 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:01.413182    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4b164646-29734020-fb8sf" event={"ID":"a79d1d5f-5dab-49a0-bb04-ed32cd858c52","Type":"ContainerDied","Data":"094caee4350742d170bcd6eb39db54e3b4859029daf043c457143f1297798832"}
Jul 14 15:00:01.414642 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:01.414615    2586 generic.go:356] "Generic (PLEG): container finished" podID="971bdfa3-4000-431a-bc2a-a88cb66599ee" containerID="6e5109f2c0d9bff1249edcd3a221e12a4b5db693425bd2b76225b7a587495047" exitCode=0
Jul 14 15:00:01.414751 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:01.414682    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734020-54d2m" event={"ID":"971bdfa3-4000-431a-bc2a-a88cb66599ee","Type":"ContainerDied","Data":"6e5109f2c0d9bff1249edcd3a221e12a4b5db693425bd2b76225b7a587495047"}
Jul 14 15:00:02.560139 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:02.560112    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4b164646-29734020-fb8sf"
Jul 14 15:00:02.594583 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:02.594558    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734020-54d2m"
Jul 14 15:00:03.426846 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:03.426816    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4b164646-29734020-fb8sf"
Jul 14 15:00:03.426846 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:03.426828    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4b164646-29734020-fb8sf" event={"ID":"a79d1d5f-5dab-49a0-bb04-ed32cd858c52","Type":"ContainerDied","Data":"dc8564933ac7d53c4ae2406c879bdd74bb52c87231c78b91bf40f59874ef5810"}
Jul 14 15:00:03.427086 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:03.426880    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc8564933ac7d53c4ae2406c879bdd74bb52c87231c78b91bf40f59874ef5810"
Jul 14 15:00:03.428333 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:03.428306    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734020-54d2m"
Jul 14 15:00:03.428485 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:03.428351    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734020-54d2m" event={"ID":"971bdfa3-4000-431a-bc2a-a88cb66599ee","Type":"ContainerDied","Data":"59edae321860b2fae22cdd0f955ac86def7ed3de501f0c2fbc2756bd2c623294"}
Jul 14 15:00:03.428485 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:03.428398    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59edae321860b2fae22cdd0f955ac86def7ed3de501f0c2fbc2756bd2c623294"
Jul 14 15:00:05.808108 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:05.808071    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-bxpj8"]
Jul 14 15:00:05.808505 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:05.808487    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a79d1d5f-5dab-49a0-bb04-ed32cd858c52" containerName="cleanup"
Jul 14 15:00:05.808505 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:05.808501    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a79d1d5f-5dab-49a0-bb04-ed32cd858c52" containerName="cleanup"
Jul 14 15:00:05.808581 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:05.808518    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971bdfa3-4000-431a-bc2a-a88cb66599ee" containerName="cleanup"
Jul 14 15:00:05.808581 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:05.808524    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="971bdfa3-4000-431a-bc2a-a88cb66599ee" containerName="cleanup"
Jul 14 15:00:05.808646 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:05.808593    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a79d1d5f-5dab-49a0-bb04-ed32cd858c52" containerName="cleanup"
Jul 14 15:00:05.808646 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:05.808604    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="971bdfa3-4000-431a-bc2a-a88cb66599ee" containerName="cleanup"
Jul 14 15:00:05.811644 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:05.811627    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-bxpj8"
Jul 14 15:00:05.814653 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:05.814634    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5695q\""
Jul 14 15:00:05.821479 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:05.821453    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-bxpj8"]
Jul 14 15:00:05.973647 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:05.973600    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761ce6c7-face-4c27-95c0-fd7af330cb84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4b164646-bxpj8\" (UID: \"761ce6c7-face-4c27-95c0-fd7af330cb84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-bxpj8"
Jul 14 15:00:06.074630 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:06.074544    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761ce6c7-face-4c27-95c0-fd7af330cb84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4b164646-bxpj8\" (UID: \"761ce6c7-face-4c27-95c0-fd7af330cb84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-bxpj8"
Jul 14 15:00:06.075114 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:06.075095    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761ce6c7-face-4c27-95c0-fd7af330cb84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4b164646-bxpj8\" (UID: \"761ce6c7-face-4c27-95c0-fd7af330cb84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-bxpj8"
Jul 14 15:00:06.124083 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:06.124045    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-bxpj8"
Jul 14 15:00:06.269265 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:06.269235    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-bxpj8"]
Jul 14 15:00:06.270764 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:00:06.270732    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761ce6c7_face_4c27_95c0_fd7af330cb84.slice/crio-7dbbfb19fc91be076de56435fbf67a1be72f1cdee21e9c403f1ee5b669c7d334 WatchSource:0}: Error finding container 7dbbfb19fc91be076de56435fbf67a1be72f1cdee21e9c403f1ee5b669c7d334: Status 404 returned error can't find the container with id 7dbbfb19fc91be076de56435fbf67a1be72f1cdee21e9c403f1ee5b669c7d334
Jul 14 15:00:06.446171 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:06.446140    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-bxpj8" event={"ID":"761ce6c7-face-4c27-95c0-fd7af330cb84","Type":"ContainerStarted","Data":"7dbbfb19fc91be076de56435fbf67a1be72f1cdee21e9c403f1ee5b669c7d334"}
Jul 14 15:00:07.453514 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:07.453479    2586 generic.go:356] "Generic (PLEG): container finished" podID="761ce6c7-face-4c27-95c0-fd7af330cb84" containerID="b657e4514e6c458043abbe48c4d4dd586987f87da407773aadff04f1631212c7" exitCode=0
Jul 14 15:00:07.453848 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:07.453563    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-bxpj8" event={"ID":"761ce6c7-face-4c27-95c0-fd7af330cb84","Type":"ContainerDied","Data":"b657e4514e6c458043abbe48c4d4dd586987f87da407773aadff04f1631212c7"}
Jul 14 15:00:08.601208 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:08.601180    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-bxpj8"
Jul 14 15:00:08.700774 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:08.700727    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761ce6c7-face-4c27-95c0-fd7af330cb84-maas-api-service-ca\") pod \"761ce6c7-face-4c27-95c0-fd7af330cb84\" (UID: \"761ce6c7-face-4c27-95c0-fd7af330cb84\") "
Jul 14 15:00:08.701107 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:08.701086    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761ce6c7-face-4c27-95c0-fd7af330cb84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "761ce6c7-face-4c27-95c0-fd7af330cb84" (UID: "761ce6c7-face-4c27-95c0-fd7af330cb84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:00:08.801479 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:08.801388    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761ce6c7-face-4c27-95c0-fd7af330cb84-maas-api-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:00:09.465702 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:09.465673    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-bxpj8"
Jul 14 15:00:09.465850 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:09.465709    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4b164646-bxpj8" event={"ID":"761ce6c7-face-4c27-95c0-fd7af330cb84","Type":"ContainerDied","Data":"7dbbfb19fc91be076de56435fbf67a1be72f1cdee21e9c403f1ee5b669c7d334"}
Jul 14 15:00:09.465850 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:09.465754    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dbbfb19fc91be076de56435fbf67a1be72f1cdee21e9c403f1ee5b669c7d334"
Jul 14 15:00:16.282779 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.282663    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"]
Jul 14 15:00:16.283308 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.283280    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp" podUID="075f5255-1fc7-4016-b525-6d0bdafdca56" containerName="maas-api" containerID="cri-o://2da4dc4cb8328f85bae92b9c46c93de4c3dd23072b7b0089e0bfb781c6c4d5a1" gracePeriod=30
Jul 14 15:00:16.310844 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.310815    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4b164646-29734020-fb8sf"]
Jul 14 15:00:16.314091 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.314067    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4b164646-29734020-fb8sf"]
Jul 14 15:00:16.504073 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.504043    2586 generic.go:356] "Generic (PLEG): container finished" podID="075f5255-1fc7-4016-b525-6d0bdafdca56" containerID="2da4dc4cb8328f85bae92b9c46c93de4c3dd23072b7b0089e0bfb781c6c4d5a1" exitCode=0
Jul 14 15:00:16.504236 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.504114    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp" event={"ID":"075f5255-1fc7-4016-b525-6d0bdafdca56","Type":"ContainerDied","Data":"2da4dc4cb8328f85bae92b9c46c93de4c3dd23072b7b0089e0bfb781c6c4d5a1"}
Jul 14 15:00:16.532332 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.532305    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 15:00:16.573211 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.573127    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64d7k\" (UniqueName: \"kubernetes.io/projected/075f5255-1fc7-4016-b525-6d0bdafdca56-kube-api-access-64d7k\") pod \"075f5255-1fc7-4016-b525-6d0bdafdca56\" (UID: \"075f5255-1fc7-4016-b525-6d0bdafdca56\") "
Jul 14 15:00:16.573211 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.573169    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075f5255-1fc7-4016-b525-6d0bdafdca56-maas-api-tls\") pod \"075f5255-1fc7-4016-b525-6d0bdafdca56\" (UID: \"075f5255-1fc7-4016-b525-6d0bdafdca56\") "
Jul 14 15:00:16.573468 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.573257    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075f5255-1fc7-4016-b525-6d0bdafdca56-openshift-service-ca\") pod \"075f5255-1fc7-4016-b525-6d0bdafdca56\" (UID: \"075f5255-1fc7-4016-b525-6d0bdafdca56\") "
Jul 14 15:00:16.573658 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.573632    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075f5255-1fc7-4016-b525-6d0bdafdca56-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "075f5255-1fc7-4016-b525-6d0bdafdca56" (UID: "075f5255-1fc7-4016-b525-6d0bdafdca56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:00:16.575312 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.575283    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075f5255-1fc7-4016-b525-6d0bdafdca56-kube-api-access-64d7k" (OuterVolumeSpecName: "kube-api-access-64d7k") pod "075f5255-1fc7-4016-b525-6d0bdafdca56" (UID: "075f5255-1fc7-4016-b525-6d0bdafdca56"). InnerVolumeSpecName "kube-api-access-64d7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:00:16.575523 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.575501    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/075f5255-1fc7-4016-b525-6d0bdafdca56-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "075f5255-1fc7-4016-b525-6d0bdafdca56" (UID: "075f5255-1fc7-4016-b525-6d0bdafdca56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:00:16.674132 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.674095    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64d7k\" (UniqueName: \"kubernetes.io/projected/075f5255-1fc7-4016-b525-6d0bdafdca56-kube-api-access-64d7k\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:00:16.674132 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.674123    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075f5255-1fc7-4016-b525-6d0bdafdca56-maas-api-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:00:16.674132 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.674133    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075f5255-1fc7-4016-b525-6d0bdafdca56-openshift-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:00:16.876319 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:16.876237    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a79d1d5f-5dab-49a0-bb04-ed32cd858c52" path="/var/lib/kubelet/pods/a79d1d5f-5dab-49a0-bb04-ed32cd858c52/volumes"
Jul 14 15:00:17.510687 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:17.510660    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"
Jul 14 15:00:17.511107 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:17.510712    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp" event={"ID":"075f5255-1fc7-4016-b525-6d0bdafdca56","Type":"ContainerDied","Data":"e804d42511769fe708c31a670ed06f12e58b7028ba364228e7b651a6b4ecd078"}
Jul 14 15:00:17.511107 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:17.510783    2586 scope.go:117] "RemoveContainer" containerID="2da4dc4cb8328f85bae92b9c46c93de4c3dd23072b7b0089e0bfb781c6c4d5a1"
Jul 14 15:00:17.529542 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:17.529514    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"]
Jul 14 15:00:17.533811 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:17.533785    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4b164646-74bbfdd487-q97bp"]
Jul 14 15:00:18.876486 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:18.876454    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="075f5255-1fc7-4016-b525-6d0bdafdca56" path="/var/lib/kubelet/pods/075f5255-1fc7-4016-b525-6d0bdafdca56/volumes"
Jul 14 15:00:22.939904 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:22.939874    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 15:00:22.940276 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:00:22.940145    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 15:01:22.998356 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:01:22.998327    2586 scope.go:117] "RemoveContainer" containerID="6c375a0a27f05854933c5d6f9bdd60db3809cc7664cfc8ba6cf037ee1d4e93ad"
Jul 14 15:02:11.448758 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.448717    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"]
Jul 14 15:02:11.449327 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.449279    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761ce6c7-face-4c27-95c0-fd7af330cb84" containerName="revoke-keys"
Jul 14 15:02:11.449327 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.449299    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="761ce6c7-face-4c27-95c0-fd7af330cb84" containerName="revoke-keys"
Jul 14 15:02:11.449477 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.449354    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="075f5255-1fc7-4016-b525-6d0bdafdca56" containerName="maas-api"
Jul 14 15:02:11.449477 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.449363    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="075f5255-1fc7-4016-b525-6d0bdafdca56" containerName="maas-api"
Jul 14 15:02:11.449477 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.449475    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="761ce6c7-face-4c27-95c0-fd7af330cb84" containerName="revoke-keys"
Jul 14 15:02:11.449629 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.449488    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="075f5255-1fc7-4016-b525-6d0bdafdca56" containerName="maas-api"
Jul 14 15:02:11.452942 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.452920    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:11.455869 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.455852    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-beb2a668-serving-cert\""
Jul 14 15:02:11.463685 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.463656    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"]
Jul 14 15:02:11.552934 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.552898    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kst7m\" (UniqueName: \"kubernetes.io/projected/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-kube-api-access-kst7m\") pod \"maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks\" (UID: \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:11.553106 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.552956    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-openshift-service-ca\") pod \"maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks\" (UID: \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:11.553106 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.552986    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-maas-api-tls\") pod \"maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks\" (UID: \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:11.654273 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.654241    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-openshift-service-ca\") pod \"maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks\" (UID: \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:11.654468 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.654284    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-maas-api-tls\") pod \"maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks\" (UID: \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:11.654468 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.654360    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kst7m\" (UniqueName: \"kubernetes.io/projected/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-kube-api-access-kst7m\") pod \"maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks\" (UID: \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:11.654562 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:02:11.654487    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-serving-cert: secret "maas-api-e2e-mt-beb2a668-serving-cert" not found
Jul 14 15:02:11.654562 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:02:11.654551    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-maas-api-tls podName:b0e587fd-0e0f-4d0d-b645-b74a41bd62a1 nodeName:}" failed. No retries permitted until 2026-07-14 15:02:12.154535972 +0000 UTC m=+1909.820722371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-maas-api-tls") pod "maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks" (UID: "b0e587fd-0e0f-4d0d-b645-b74a41bd62a1") : secret "maas-api-e2e-mt-beb2a668-serving-cert" not found
Jul 14 15:02:11.655142 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.655111    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-openshift-service-ca\") pod \"maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks\" (UID: \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:11.665288 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:11.665265    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kst7m\" (UniqueName: \"kubernetes.io/projected/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-kube-api-access-kst7m\") pod \"maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks\" (UID: \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:12.158086 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:12.158055    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-maas-api-tls\") pod \"maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks\" (UID: \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:12.160582 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:12.160563    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-maas-api-tls\") pod \"maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks\" (UID: \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:12.364675 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:12.364634    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:12.528602 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:12.528568    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"]
Jul 14 15:02:12.532062 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:02:12.532035    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e587fd_0e0f_4d0d_b645_b74a41bd62a1.slice/crio-a07c55876f4ff10507325025c769b69fbdc0f44004e1c3c666605ad4d08ac696 WatchSource:0}: Error finding container a07c55876f4ff10507325025c769b69fbdc0f44004e1c3c666605ad4d08ac696: Status 404 returned error can't find the container with id a07c55876f4ff10507325025c769b69fbdc0f44004e1c3c666605ad4d08ac696
Jul 14 15:02:12.533342 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:12.533321    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 15:02:13.105629 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:13.105591    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks" event={"ID":"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1","Type":"ContainerStarted","Data":"a07c55876f4ff10507325025c769b69fbdc0f44004e1c3c666605ad4d08ac696"}
Jul 14 15:02:14.112695 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:14.112653    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks" event={"ID":"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1","Type":"ContainerStarted","Data":"b03beb0613e078c9fccf4ea0cf40c4385c60a4bbba090b2b100ccc77792a702b"}
Jul 14 15:02:14.113065 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:14.112794    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:14.132341 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:14.132292    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks" podStartSLOduration=2.138116565 podStartE2EDuration="3.132277843s" totalImagesPullingTime="994.161278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:02:14.130199144 +0000 UTC m=+1911.796385565" watchObservedRunningTime="2026-07-14 15:02:14.132277843 +0000 UTC m=+1911.798464264"
Jul 14 15:02:20.123920 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:20.123887    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:22.927980 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:22.927945    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5"]
Jul 14 15:02:22.931523 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:22.931497    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5"
Jul 14 15:02:22.934484 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:22.934462    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5695q\""
Jul 14 15:02:22.940965 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:22.940877    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5"]
Jul 14 15:02:22.958803 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:22.957172    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-beb2a668-q77r5\" (UID: \"e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5"
Jul 14 15:02:23.058360 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:23.058327    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-beb2a668-q77r5\" (UID: \"e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5"
Jul 14 15:02:23.058944 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:23.058922    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-beb2a668-q77r5\" (UID: \"e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5"
Jul 14 15:02:23.243600 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:23.243510    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5"
Jul 14 15:02:23.393802 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:23.393725    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5"]
Jul 14 15:02:23.396150 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:02:23.396112    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0238dda_d3c0_4c7f_8b6f_6c2aa4acb560.slice/crio-1052c5f385d3370131cf26080d027030ab5dac4ddc4194f6c5d6153e1fe716a4 WatchSource:0}: Error finding container 1052c5f385d3370131cf26080d027030ab5dac4ddc4194f6c5d6153e1fe716a4: Status 404 returned error can't find the container with id 1052c5f385d3370131cf26080d027030ab5dac4ddc4194f6c5d6153e1fe716a4
Jul 14 15:02:24.166242 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:24.166143    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5" event={"ID":"e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560","Type":"ContainerStarted","Data":"df11beb7b3972b32b4a6a9d4cedc2b0656d9aad2e02c5ee6092de805e1dd7d90"}
Jul 14 15:02:24.166242 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:24.166199    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5" event={"ID":"e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560","Type":"ContainerStarted","Data":"1052c5f385d3370131cf26080d027030ab5dac4ddc4194f6c5d6153e1fe716a4"}
Jul 14 15:02:24.183801 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:24.183737    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5" podStartSLOduration=2.183716721 podStartE2EDuration="2.183716721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:02:24.181825487 +0000 UTC m=+1921.848011905" watchObservedRunningTime="2026-07-14 15:02:24.183716721 +0000 UTC m=+1921.849903145"
Jul 14 15:02:25.172027 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:25.171993    2586 generic.go:356] "Generic (PLEG): container finished" podID="e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560" containerID="df11beb7b3972b32b4a6a9d4cedc2b0656d9aad2e02c5ee6092de805e1dd7d90" exitCode=0
Jul 14 15:02:25.172397 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:25.172082    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5" event={"ID":"e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560","Type":"ContainerDied","Data":"df11beb7b3972b32b4a6a9d4cedc2b0656d9aad2e02c5ee6092de805e1dd7d90"}
Jul 14 15:02:26.311451 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:26.311425    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5"
Jul 14 15:02:26.388727 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:26.388692    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560-maas-api-service-ca\") pod \"e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560\" (UID: \"e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560\") "
Jul 14 15:02:26.389073 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:26.389051    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560" (UID: "e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:02:26.489979 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:26.489895    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560-maas-api-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:02:27.183908 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:27.183879    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5"
Jul 14 15:02:27.183908 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:27.183899    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-beb2a668-q77r5" event={"ID":"e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560","Type":"ContainerDied","Data":"1052c5f385d3370131cf26080d027030ab5dac4ddc4194f6c5d6153e1fe716a4"}
Jul 14 15:02:27.184119 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:27.183952    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1052c5f385d3370131cf26080d027030ab5dac4ddc4194f6c5d6153e1fe716a4"
Jul 14 15:02:33.401183 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:33.401147    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"]
Jul 14 15:02:33.402194 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:33.402144    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks" podUID="b0e587fd-0e0f-4d0d-b645-b74a41bd62a1" containerName="maas-api" containerID="cri-o://b03beb0613e078c9fccf4ea0cf40c4385c60a4bbba090b2b100ccc77792a702b" gracePeriod=30
Jul 14 15:02:33.644208 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:33.644181    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:33.758812 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:33.758775    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-openshift-service-ca\") pod \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\" (UID: \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\") "
Jul 14 15:02:33.758992 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:33.758819    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-maas-api-tls\") pod \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\" (UID: \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\") "
Jul 14 15:02:33.758992 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:33.758862    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kst7m\" (UniqueName: \"kubernetes.io/projected/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-kube-api-access-kst7m\") pod \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\" (UID: \"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1\") "
Jul 14 15:02:33.759169 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:33.759145    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0e587fd-0e0f-4d0d-b645-b74a41bd62a1" (UID: "b0e587fd-0e0f-4d0d-b645-b74a41bd62a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:02:33.761155 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:33.761133    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0e587fd-0e0f-4d0d-b645-b74a41bd62a1" (UID: "b0e587fd-0e0f-4d0d-b645-b74a41bd62a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:02:33.761262 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:33.761195    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-kube-api-access-kst7m" (OuterVolumeSpecName: "kube-api-access-kst7m") pod "b0e587fd-0e0f-4d0d-b645-b74a41bd62a1" (UID: "b0e587fd-0e0f-4d0d-b645-b74a41bd62a1"). InnerVolumeSpecName "kube-api-access-kst7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:02:33.859746 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:33.859707    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kst7m\" (UniqueName: \"kubernetes.io/projected/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-kube-api-access-kst7m\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:02:33.859746 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:33.859740    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-openshift-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:02:33.859746 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:33.859750    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1-maas-api-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:02:34.223451 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:34.223394    2586 generic.go:356] "Generic (PLEG): container finished" podID="b0e587fd-0e0f-4d0d-b645-b74a41bd62a1" containerID="b03beb0613e078c9fccf4ea0cf40c4385c60a4bbba090b2b100ccc77792a702b" exitCode=0
Jul 14 15:02:34.223451 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:34.223449    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"
Jul 14 15:02:34.223693 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:34.223477    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks" event={"ID":"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1","Type":"ContainerDied","Data":"b03beb0613e078c9fccf4ea0cf40c4385c60a4bbba090b2b100ccc77792a702b"}
Jul 14 15:02:34.223693 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:34.223535    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks" event={"ID":"b0e587fd-0e0f-4d0d-b645-b74a41bd62a1","Type":"ContainerDied","Data":"a07c55876f4ff10507325025c769b69fbdc0f44004e1c3c666605ad4d08ac696"}
Jul 14 15:02:34.223693 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:34.223557    2586 scope.go:117] "RemoveContainer" containerID="b03beb0613e078c9fccf4ea0cf40c4385c60a4bbba090b2b100ccc77792a702b"
Jul 14 15:02:34.233177 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:34.233162    2586 scope.go:117] "RemoveContainer" containerID="b03beb0613e078c9fccf4ea0cf40c4385c60a4bbba090b2b100ccc77792a702b"
Jul 14 15:02:34.233458 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:02:34.233439    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03beb0613e078c9fccf4ea0cf40c4385c60a4bbba090b2b100ccc77792a702b\": container with ID starting with b03beb0613e078c9fccf4ea0cf40c4385c60a4bbba090b2b100ccc77792a702b not found: ID does not exist" containerID="b03beb0613e078c9fccf4ea0cf40c4385c60a4bbba090b2b100ccc77792a702b"
Jul 14 15:02:34.233527 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:34.233470    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03beb0613e078c9fccf4ea0cf40c4385c60a4bbba090b2b100ccc77792a702b"} err="failed to get container status \"b03beb0613e078c9fccf4ea0cf40c4385c60a4bbba090b2b100ccc77792a702b\": rpc error: code = NotFound desc = could not find container \"b03beb0613e078c9fccf4ea0cf40c4385c60a4bbba090b2b100ccc77792a702b\": container with ID starting with b03beb0613e078c9fccf4ea0cf40c4385c60a4bbba090b2b100ccc77792a702b not found: ID does not exist"
Jul 14 15:02:34.249468 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:34.249444    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"]
Jul 14 15:02:34.256719 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:34.256697    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-beb2a668-b659d7cbd-bd9ks"]
Jul 14 15:02:34.876037 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:02:34.876001    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0e587fd-0e0f-4d0d-b645-b74a41bd62a1" path="/var/lib/kubelet/pods/b0e587fd-0e0f-4d0d-b645-b74a41bd62a1/volumes"
Jul 14 15:03:11.718056 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.718020    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"]
Jul 14 15:03:11.718631 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.718546    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0e587fd-0e0f-4d0d-b645-b74a41bd62a1" containerName="maas-api"
Jul 14 15:03:11.718631 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.718561    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0e587fd-0e0f-4d0d-b645-b74a41bd62a1" containerName="maas-api"
Jul 14 15:03:11.718631 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.718577    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560" containerName="revoke-keys"
Jul 14 15:03:11.718631 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.718586    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560" containerName="revoke-keys"
Jul 14 15:03:11.718832 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.718650    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0e587fd-0e0f-4d0d-b645-b74a41bd62a1" containerName="maas-api"
Jul 14 15:03:11.718832 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.718662    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560" containerName="revoke-keys"
Jul 14 15:03:11.722444 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.722403    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:03:11.725359 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.725338    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3fe6f3d4-serving-cert\""
Jul 14 15:03:11.741274 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.741250    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"]
Jul 14 15:03:11.810316 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.810286    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs5zz\" (UniqueName: \"kubernetes.io/projected/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-kube-api-access-rs5zz\") pod \"maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq\" (UID: \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:03:11.810498 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.810344    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-openshift-service-ca\") pod \"maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq\" (UID: \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:03:11.810498 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.810483    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-maas-api-tls\") pod \"maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq\" (UID: \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:03:11.911505 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.911473    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs5zz\" (UniqueName: \"kubernetes.io/projected/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-kube-api-access-rs5zz\") pod \"maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq\" (UID: \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:03:11.911671 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.911615    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-openshift-service-ca\") pod \"maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq\" (UID: \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:03:11.911721 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.911697    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-maas-api-tls\") pod \"maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq\" (UID: \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:03:11.911807 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:03:11.911794    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-serving-cert: secret "maas-api-e2e-mt-3fe6f3d4-serving-cert" not found
Jul 14 15:03:11.911855 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:03:11.911848    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-maas-api-tls podName:d16ea391-c8a4-43b8-8bc2-d110e1ff4254 nodeName:}" failed. No retries permitted until 2026-07-14 15:03:12.411832796 +0000 UTC m=+1970.078019196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-maas-api-tls") pod "maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq" (UID: "d16ea391-c8a4-43b8-8bc2-d110e1ff4254") : secret "maas-api-e2e-mt-3fe6f3d4-serving-cert" not found
Jul 14 15:03:11.912254 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.912235    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-openshift-service-ca\") pod \"maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq\" (UID: \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:03:11.925532 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:11.925503    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs5zz\" (UniqueName: \"kubernetes.io/projected/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-kube-api-access-rs5zz\") pod \"maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq\" (UID: \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:03:12.417093 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:12.417054    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-maas-api-tls\") pod \"maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq\" (UID: \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:03:12.419954 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:12.419929    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-maas-api-tls\") pod \"maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq\" (UID: \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:03:12.632521 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:12.632488    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:03:12.771195 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:12.771169    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"]
Jul 14 15:03:12.773033 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:03:12.773006    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16ea391_c8a4_43b8_8bc2_d110e1ff4254.slice/crio-e675aaf1339e496928e0cbc475e637b7e5bbd7e4a656e7f4342e9d96f5c4d0b8 WatchSource:0}: Error finding container e675aaf1339e496928e0cbc475e637b7e5bbd7e4a656e7f4342e9d96f5c4d0b8: Status 404 returned error can't find the container with id e675aaf1339e496928e0cbc475e637b7e5bbd7e4a656e7f4342e9d96f5c4d0b8
Jul 14 15:03:13.439372 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:13.439330    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq" event={"ID":"d16ea391-c8a4-43b8-8bc2-d110e1ff4254","Type":"ContainerStarted","Data":"e675aaf1339e496928e0cbc475e637b7e5bbd7e4a656e7f4342e9d96f5c4d0b8"}
Jul 14 15:03:14.445985 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:14.445945    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq" event={"ID":"d16ea391-c8a4-43b8-8bc2-d110e1ff4254","Type":"ContainerStarted","Data":"d6fdc226d44baccfb45d617e4a10713b28ed56c41007d3e090552b6e21197b92"}
Jul 14 15:03:14.446375 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:14.446041    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:03:14.467625 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:14.467578    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq" podStartSLOduration=2.400131337 podStartE2EDuration="3.467563326s" totalImagesPullingTime="1.067431989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:03:14.466236208 +0000 UTC m=+1972.132422628" watchObservedRunningTime="2026-07-14 15:03:14.467563326 +0000 UTC m=+1972.133749747"
Jul 14 15:03:20.456612 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:20.456580    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:03:23.026456 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:23.026398    2586 scope.go:117] "RemoveContainer" containerID="98675606ceb9c0e7c582af05559b45e723cbd189973347dbe52cd844b0065876"
Jul 14 15:03:32.758485 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:32.758446    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"]
Jul 14 15:03:32.762167 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:32.762147    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:32.772459 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:32.772434    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3fe6f3d4\"/\"openshift-service-ca.crt\""
Jul 14 15:03:32.772459 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:32.772456    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3fe6f3d4\"/\"e2e-lifecycle-model-3fe6f3d4-kserve-self-signed-certs\""
Jul 14 15:03:32.772632 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:32.772438    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3fe6f3d4\"/\"kube-root-ca.crt\""
Jul 14 15:03:32.772632 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:32.772438    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3fe6f3d4\"/\"default-dockercfg-r6gbs\""
Jul 14 15:03:32.778596 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:32.778573    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"]
Jul 14 15:03:32.911199 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:32.911162    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-model-cache\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:32.911390 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:32.911220    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-dshm\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:32.911390 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:32.911274    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-home\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:32.911390 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:32.911316    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0888d6df-d6f9-4670-af46-78d03e2483d0-tls-certs\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:32.911390 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:32.911340    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wsn\" (UniqueName: \"kubernetes.io/projected/0888d6df-d6f9-4670-af46-78d03e2483d0-kube-api-access-85wsn\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:32.911390 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:32.911361    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-tmp-dir\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:33.012204 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.012110    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-dshm\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:33.012204 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.012165    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-home\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:33.012442 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.012208    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0888d6df-d6f9-4670-af46-78d03e2483d0-tls-certs\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:33.012442 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.012235    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wsn\" (UniqueName: \"kubernetes.io/projected/0888d6df-d6f9-4670-af46-78d03e2483d0-kube-api-access-85wsn\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:33.012442 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.012257    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-tmp-dir\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:33.012442 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.012273    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-model-cache\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:33.012708 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.012679    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-home\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:33.012835 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.012729    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-tmp-dir\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:33.012835 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.012776    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-model-cache\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:33.014698 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.014672    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-dshm\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:33.014958 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.014939    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0888d6df-d6f9-4670-af46-78d03e2483d0-tls-certs\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:33.021679 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.021659    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wsn\" (UniqueName: \"kubernetes.io/projected/0888d6df-d6f9-4670-af46-78d03e2483d0-kube-api-access-85wsn\") pod \"e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") " pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:33.071568 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.071537    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:33.215510 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.215479    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"]
Jul 14 15:03:33.216642 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:03:33.216616    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0888d6df_d6f9_4670_af46_78d03e2483d0.slice/crio-7aeeb3235169c318a7528cc07dc82c7fe7adad6def55de642104197c0e627c26 WatchSource:0}: Error finding container 7aeeb3235169c318a7528cc07dc82c7fe7adad6def55de642104197c0e627c26: Status 404 returned error can't find the container with id 7aeeb3235169c318a7528cc07dc82c7fe7adad6def55de642104197c0e627c26
Jul 14 15:03:33.550818 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.550709    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r" event={"ID":"0888d6df-d6f9-4670-af46-78d03e2483d0","Type":"ContainerStarted","Data":"372cc06788f92c554f0ad218a6bc946cb3cbcab93326e7b9fd90d3c55c6a77eb"}
Jul 14 15:03:33.550818 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.550754    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r" event={"ID":"0888d6df-d6f9-4670-af46-78d03e2483d0","Type":"ContainerStarted","Data":"7aeeb3235169c318a7528cc07dc82c7fe7adad6def55de642104197c0e627c26"}
Jul 14 15:03:33.575650 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:33.575595    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r" podStartSLOduration=1.575578888 podStartE2EDuration="1.575578888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:03:33.574731247 +0000 UTC m=+1991.240917668" watchObservedRunningTime="2026-07-14 15:03:33.575578888 +0000 UTC m=+1991.241765308"
Jul 14 15:03:43.072288 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:43.072245    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:43.072288 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:43.072288    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:43.084994 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:43.084963    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:03:43.616994 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:03:43.616964    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:04:09.843868 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:09.843818    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"]
Jul 14 15:04:09.844336 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:09.844082    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq" podUID="d16ea391-c8a4-43b8-8bc2-d110e1ff4254" containerName="maas-api" containerID="cri-o://d6fdc226d44baccfb45d617e4a10713b28ed56c41007d3e090552b6e21197b92" gracePeriod=30
Jul 14 15:04:10.097840 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.097781    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:04:10.270039 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.270003    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-maas-api-tls\") pod \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\" (UID: \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\") "
Jul 14 15:04:10.270228 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.270055    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs5zz\" (UniqueName: \"kubernetes.io/projected/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-kube-api-access-rs5zz\") pod \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\" (UID: \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\") "
Jul 14 15:04:10.270228 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.270086    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-openshift-service-ca\") pod \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\" (UID: \"d16ea391-c8a4-43b8-8bc2-d110e1ff4254\") "
Jul 14 15:04:10.270547 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.270521    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d16ea391-c8a4-43b8-8bc2-d110e1ff4254" (UID: "d16ea391-c8a4-43b8-8bc2-d110e1ff4254"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:04:10.272291 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.272267    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d16ea391-c8a4-43b8-8bc2-d110e1ff4254" (UID: "d16ea391-c8a4-43b8-8bc2-d110e1ff4254"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:04:10.272389 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.272301    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-kube-api-access-rs5zz" (OuterVolumeSpecName: "kube-api-access-rs5zz") pod "d16ea391-c8a4-43b8-8bc2-d110e1ff4254" (UID: "d16ea391-c8a4-43b8-8bc2-d110e1ff4254"). InnerVolumeSpecName "kube-api-access-rs5zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:04:10.370999 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.370909    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-maas-api-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:04:10.370999 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.370939    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs5zz\" (UniqueName: \"kubernetes.io/projected/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-kube-api-access-rs5zz\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:04:10.370999 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.370950    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16ea391-c8a4-43b8-8bc2-d110e1ff4254-openshift-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:04:10.754012 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.753975    2586 generic.go:356] "Generic (PLEG): container finished" podID="d16ea391-c8a4-43b8-8bc2-d110e1ff4254" containerID="d6fdc226d44baccfb45d617e4a10713b28ed56c41007d3e090552b6e21197b92" exitCode=0
Jul 14 15:04:10.754187 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.754026    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"
Jul 14 15:04:10.754187 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.754038    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq" event={"ID":"d16ea391-c8a4-43b8-8bc2-d110e1ff4254","Type":"ContainerDied","Data":"d6fdc226d44baccfb45d617e4a10713b28ed56c41007d3e090552b6e21197b92"}
Jul 14 15:04:10.754187 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.754080    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq" event={"ID":"d16ea391-c8a4-43b8-8bc2-d110e1ff4254","Type":"ContainerDied","Data":"e675aaf1339e496928e0cbc475e637b7e5bbd7e4a656e7f4342e9d96f5c4d0b8"}
Jul 14 15:04:10.754187 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.754102    2586 scope.go:117] "RemoveContainer" containerID="d6fdc226d44baccfb45d617e4a10713b28ed56c41007d3e090552b6e21197b92"
Jul 14 15:04:10.763465 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.763445    2586 scope.go:117] "RemoveContainer" containerID="d6fdc226d44baccfb45d617e4a10713b28ed56c41007d3e090552b6e21197b92"
Jul 14 15:04:10.763746 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:04:10.763729    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6fdc226d44baccfb45d617e4a10713b28ed56c41007d3e090552b6e21197b92\": container with ID starting with d6fdc226d44baccfb45d617e4a10713b28ed56c41007d3e090552b6e21197b92 not found: ID does not exist" containerID="d6fdc226d44baccfb45d617e4a10713b28ed56c41007d3e090552b6e21197b92"
Jul 14 15:04:10.763815 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.763754    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fdc226d44baccfb45d617e4a10713b28ed56c41007d3e090552b6e21197b92"} err="failed to get container status \"d6fdc226d44baccfb45d617e4a10713b28ed56c41007d3e090552b6e21197b92\": rpc error: code = NotFound desc = could not find container \"d6fdc226d44baccfb45d617e4a10713b28ed56c41007d3e090552b6e21197b92\": container with ID starting with d6fdc226d44baccfb45d617e4a10713b28ed56c41007d3e090552b6e21197b92 not found: ID does not exist"
Jul 14 15:04:10.777026 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.776993    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"]
Jul 14 15:04:10.781311 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.781288    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe6f3d4-7f46d44b85-x4nlq"]
Jul 14 15:04:10.876479 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:10.876447    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d16ea391-c8a4-43b8-8bc2-d110e1ff4254" path="/var/lib/kubelet/pods/d16ea391-c8a4-43b8-8bc2-d110e1ff4254/volumes"
Jul 14 15:04:19.468165 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:19.468124    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"]
Jul 14 15:04:23.050527 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:23.050488    2586 scope.go:117] "RemoveContainer" containerID="234962364fbc1822ff45a6f9cd45526cbc2c9548f82db3f26e58109592564885"
Jul 14 15:04:34.495377 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.495323    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r" podUID="0888d6df-d6f9-4670-af46-78d03e2483d0" containerName="main" containerID="cri-o://372cc06788f92c554f0ad218a6bc946cb3cbcab93326e7b9fd90d3c55c6a77eb" gracePeriod=45
Jul 14 15:04:34.746505 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.746440    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:04:34.792476 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.792447    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85wsn\" (UniqueName: \"kubernetes.io/projected/0888d6df-d6f9-4670-af46-78d03e2483d0-kube-api-access-85wsn\") pod \"0888d6df-d6f9-4670-af46-78d03e2483d0\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") "
Jul 14 15:04:34.792658 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.792495    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-dshm\") pod \"0888d6df-d6f9-4670-af46-78d03e2483d0\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") "
Jul 14 15:04:34.792658 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.792542    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-tmp-dir\") pod \"0888d6df-d6f9-4670-af46-78d03e2483d0\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") "
Jul 14 15:04:34.792658 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.792597    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-model-cache\") pod \"0888d6df-d6f9-4670-af46-78d03e2483d0\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") "
Jul 14 15:04:34.792658 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.792633    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0888d6df-d6f9-4670-af46-78d03e2483d0-tls-certs\") pod \"0888d6df-d6f9-4670-af46-78d03e2483d0\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") "
Jul 14 15:04:34.792877 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.792679    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-home\") pod \"0888d6df-d6f9-4670-af46-78d03e2483d0\" (UID: \"0888d6df-d6f9-4670-af46-78d03e2483d0\") "
Jul 14 15:04:34.792933 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.792882    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0888d6df-d6f9-4670-af46-78d03e2483d0" (UID: "0888d6df-d6f9-4670-af46-78d03e2483d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:04:34.793031 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.792999    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-tmp-dir\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:04:34.793186 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.793059    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-home" (OuterVolumeSpecName: "home") pod "0888d6df-d6f9-4670-af46-78d03e2483d0" (UID: "0888d6df-d6f9-4670-af46-78d03e2483d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:04:34.793186 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.793110    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "0888d6df-d6f9-4670-af46-78d03e2483d0" (UID: "0888d6df-d6f9-4670-af46-78d03e2483d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:04:34.794988 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.794959    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0888d6df-d6f9-4670-af46-78d03e2483d0-kube-api-access-85wsn" (OuterVolumeSpecName: "kube-api-access-85wsn") pod "0888d6df-d6f9-4670-af46-78d03e2483d0" (UID: "0888d6df-d6f9-4670-af46-78d03e2483d0"). InnerVolumeSpecName "kube-api-access-85wsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:04:34.795166 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.795145    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-dshm" (OuterVolumeSpecName: "dshm") pod "0888d6df-d6f9-4670-af46-78d03e2483d0" (UID: "0888d6df-d6f9-4670-af46-78d03e2483d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:04:34.795331 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.795312    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0888d6df-d6f9-4670-af46-78d03e2483d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0888d6df-d6f9-4670-af46-78d03e2483d0" (UID: "0888d6df-d6f9-4670-af46-78d03e2483d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:04:34.879641 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.879601    2586 generic.go:356] "Generic (PLEG): container finished" podID="0888d6df-d6f9-4670-af46-78d03e2483d0" containerID="372cc06788f92c554f0ad218a6bc946cb3cbcab93326e7b9fd90d3c55c6a77eb" exitCode=0
Jul 14 15:04:34.879641 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.879645    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"
Jul 14 15:04:34.879899 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.879684    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r" event={"ID":"0888d6df-d6f9-4670-af46-78d03e2483d0","Type":"ContainerDied","Data":"372cc06788f92c554f0ad218a6bc946cb3cbcab93326e7b9fd90d3c55c6a77eb"}
Jul 14 15:04:34.879899 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.879730    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r" event={"ID":"0888d6df-d6f9-4670-af46-78d03e2483d0","Type":"ContainerDied","Data":"7aeeb3235169c318a7528cc07dc82c7fe7adad6def55de642104197c0e627c26"}
Jul 14 15:04:34.879899 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.879752    2586 scope.go:117] "RemoveContainer" containerID="372cc06788f92c554f0ad218a6bc946cb3cbcab93326e7b9fd90d3c55c6a77eb"
Jul 14 15:04:34.889777 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.889759    2586 scope.go:117] "RemoveContainer" containerID="372cc06788f92c554f0ad218a6bc946cb3cbcab93326e7b9fd90d3c55c6a77eb"
Jul 14 15:04:34.890086 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:04:34.890067    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372cc06788f92c554f0ad218a6bc946cb3cbcab93326e7b9fd90d3c55c6a77eb\": container with ID starting with 372cc06788f92c554f0ad218a6bc946cb3cbcab93326e7b9fd90d3c55c6a77eb not found: ID does not exist" containerID="372cc06788f92c554f0ad218a6bc946cb3cbcab93326e7b9fd90d3c55c6a77eb"
Jul 14 15:04:34.890144 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.890094    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372cc06788f92c554f0ad218a6bc946cb3cbcab93326e7b9fd90d3c55c6a77eb"} err="failed to get container status \"372cc06788f92c554f0ad218a6bc946cb3cbcab93326e7b9fd90d3c55c6a77eb\": rpc error: code = NotFound desc = could not find container \"372cc06788f92c554f0ad218a6bc946cb3cbcab93326e7b9fd90d3c55c6a77eb\": container with ID starting with 372cc06788f92c554f0ad218a6bc946cb3cbcab93326e7b9fd90d3c55c6a77eb not found: ID does not exist"
Jul 14 15:04:34.894450 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.894431    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85wsn\" (UniqueName: \"kubernetes.io/projected/0888d6df-d6f9-4670-af46-78d03e2483d0-kube-api-access-85wsn\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:04:34.894503 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.894452    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-dshm\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:04:34.894503 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.894461    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-model-cache\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:04:34.894503 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.894471    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0888d6df-d6f9-4670-af46-78d03e2483d0-tls-certs\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:04:34.894503 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.894479    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0888d6df-d6f9-4670-af46-78d03e2483d0-home\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:04:34.901206 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.901182    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"]
Jul 14 15:04:34.905609 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:34.905588    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3fe6f3d4/e2e-lifecycle-model-3fe6f3d4-kserve-5bcc9cb767-9f78r"]
Jul 14 15:04:36.876669 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:04:36.876636    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0888d6df-d6f9-4670-af46-78d03e2483d0" path="/var/lib/kubelet/pods/0888d6df-d6f9-4670-af46-78d03e2483d0/volumes"
Jul 14 15:05:22.977089 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:05:22.977059    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 15:05:22.979072 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:05:22.979050    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 15:06:04.555946 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.555913    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"]
Jul 14 15:06:04.556454 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.556330    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d16ea391-c8a4-43b8-8bc2-d110e1ff4254" containerName="maas-api"
Jul 14 15:06:04.556454 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.556357    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d16ea391-c8a4-43b8-8bc2-d110e1ff4254" containerName="maas-api"
Jul 14 15:06:04.556454 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.556382    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0888d6df-d6f9-4670-af46-78d03e2483d0" containerName="main"
Jul 14 15:06:04.556454 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.556390    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0888d6df-d6f9-4670-af46-78d03e2483d0" containerName="main"
Jul 14 15:06:04.556613 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.556506    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d16ea391-c8a4-43b8-8bc2-d110e1ff4254" containerName="maas-api"
Jul 14 15:06:04.556613 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.556527    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0888d6df-d6f9-4670-af46-78d03e2483d0" containerName="main"
Jul 14 15:06:04.559587 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.559570    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:06:04.562878 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.562859    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-03dca1-serving-cert\""
Jul 14 15:06:04.575139 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.575116    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"]
Jul 14 15:06:04.624617 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.624585    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0252fa82-9c5d-4216-ab03-487d51675632-maas-api-tls\") pod \"maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv\" (UID: \"0252fa82-9c5d-4216-ab03-487d51675632\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:06:04.624813 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.624665    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0252fa82-9c5d-4216-ab03-487d51675632-openshift-service-ca\") pod \"maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv\" (UID: \"0252fa82-9c5d-4216-ab03-487d51675632\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:06:04.624813 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.624735    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx26c\" (UniqueName: \"kubernetes.io/projected/0252fa82-9c5d-4216-ab03-487d51675632-kube-api-access-mx26c\") pod \"maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv\" (UID: \"0252fa82-9c5d-4216-ab03-487d51675632\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:06:04.726208 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.726171    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx26c\" (UniqueName: \"kubernetes.io/projected/0252fa82-9c5d-4216-ab03-487d51675632-kube-api-access-mx26c\") pod \"maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv\" (UID: \"0252fa82-9c5d-4216-ab03-487d51675632\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:06:04.726378 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.726232    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0252fa82-9c5d-4216-ab03-487d51675632-maas-api-tls\") pod \"maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv\" (UID: \"0252fa82-9c5d-4216-ab03-487d51675632\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:06:04.726378 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.726268    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0252fa82-9c5d-4216-ab03-487d51675632-openshift-service-ca\") pod \"maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv\" (UID: \"0252fa82-9c5d-4216-ab03-487d51675632\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:06:04.726934 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.726909    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0252fa82-9c5d-4216-ab03-487d51675632-openshift-service-ca\") pod \"maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv\" (UID: \"0252fa82-9c5d-4216-ab03-487d51675632\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:06:04.728828 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.728811    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0252fa82-9c5d-4216-ab03-487d51675632-maas-api-tls\") pod \"maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv\" (UID: \"0252fa82-9c5d-4216-ab03-487d51675632\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:06:04.735748 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.735725    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx26c\" (UniqueName: \"kubernetes.io/projected/0252fa82-9c5d-4216-ab03-487d51675632-kube-api-access-mx26c\") pod \"maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv\" (UID: \"0252fa82-9c5d-4216-ab03-487d51675632\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:06:04.869525 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:04.869427    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:06:05.014333 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:05.014306    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"]
Jul 14 15:06:05.015264 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:06:05.015243    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252fa82_9c5d_4216_ab03_487d51675632.slice/crio-01133f37520cb863c5caec306eb7978d5a15e80f894a62761c592c0ec7e7f597 WatchSource:0}: Error finding container 01133f37520cb863c5caec306eb7978d5a15e80f894a62761c592c0ec7e7f597: Status 404 returned error can't find the container with id 01133f37520cb863c5caec306eb7978d5a15e80f894a62761c592c0ec7e7f597
Jul 14 15:06:05.345280 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:05.345244    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv" event={"ID":"0252fa82-9c5d-4216-ab03-487d51675632","Type":"ContainerStarted","Data":"01133f37520cb863c5caec306eb7978d5a15e80f894a62761c592c0ec7e7f597"}
Jul 14 15:06:06.352472 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:06.352439    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv" event={"ID":"0252fa82-9c5d-4216-ab03-487d51675632","Type":"ContainerStarted","Data":"7ae1702d0514ff3de0c5df13ed870325979636d6463fde2706ad7c0a359be828"}
Jul 14 15:06:06.352854 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:06.352563    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:06:06.372508 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:06.372454    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv" podStartSLOduration=1.248166442 podStartE2EDuration="2.372435678s" totalImagesPullingTime="1.124269236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:06:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:06:06.370318239 +0000 UTC m=+2144.036504661" watchObservedRunningTime="2026-07-14 15:06:06.372435678 +0000 UTC m=+2144.038622097"
Jul 14 15:06:12.362713 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.362679    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:06:12.590263 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.590225    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"]
Jul 14 15:06:12.594934 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.594911    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:06:12.598289 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.598265    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4226a2-serving-cert\""
Jul 14 15:06:12.616579 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.616495    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"]
Jul 14 15:06:12.708965 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.708925    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af5e9828-22f9-4bd0-aeb8-45960410ced7-maas-api-tls\") pod \"maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2\" (UID: \"af5e9828-22f9-4bd0-aeb8-45960410ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:06:12.709133 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.708972    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz55m\" (UniqueName: \"kubernetes.io/projected/af5e9828-22f9-4bd0-aeb8-45960410ced7-kube-api-access-bz55m\") pod \"maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2\" (UID: \"af5e9828-22f9-4bd0-aeb8-45960410ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:06:12.709133 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.709049    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af5e9828-22f9-4bd0-aeb8-45960410ced7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2\" (UID: \"af5e9828-22f9-4bd0-aeb8-45960410ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:06:12.810568 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.810528    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af5e9828-22f9-4bd0-aeb8-45960410ced7-maas-api-tls\") pod \"maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2\" (UID: \"af5e9828-22f9-4bd0-aeb8-45960410ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:06:12.810568 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.810571    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz55m\" (UniqueName: \"kubernetes.io/projected/af5e9828-22f9-4bd0-aeb8-45960410ced7-kube-api-access-bz55m\") pod \"maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2\" (UID: \"af5e9828-22f9-4bd0-aeb8-45960410ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:06:12.810764 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.810621    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af5e9828-22f9-4bd0-aeb8-45960410ced7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2\" (UID: \"af5e9828-22f9-4bd0-aeb8-45960410ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:06:12.811299 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.811280    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af5e9828-22f9-4bd0-aeb8-45960410ced7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2\" (UID: \"af5e9828-22f9-4bd0-aeb8-45960410ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:06:12.813056 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.813037    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af5e9828-22f9-4bd0-aeb8-45960410ced7-maas-api-tls\") pod \"maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2\" (UID: \"af5e9828-22f9-4bd0-aeb8-45960410ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:06:12.819814 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.819790    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz55m\" (UniqueName: \"kubernetes.io/projected/af5e9828-22f9-4bd0-aeb8-45960410ced7-kube-api-access-bz55m\") pod \"maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2\" (UID: \"af5e9828-22f9-4bd0-aeb8-45960410ced7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:06:12.913210 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:12.913123    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:06:13.058299 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:13.058272    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"]
Jul 14 15:06:13.060379 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:06:13.060352    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5e9828_22f9_4bd0_aeb8_45960410ced7.slice/crio-0c68471118e1c42ee801257599d83867dabab9ac03b203642c468aa6f0039463 WatchSource:0}: Error finding container 0c68471118e1c42ee801257599d83867dabab9ac03b203642c468aa6f0039463: Status 404 returned error can't find the container with id 0c68471118e1c42ee801257599d83867dabab9ac03b203642c468aa6f0039463
Jul 14 15:06:13.389196 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:13.389158    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2" event={"ID":"af5e9828-22f9-4bd0-aeb8-45960410ced7","Type":"ContainerStarted","Data":"0c68471118e1c42ee801257599d83867dabab9ac03b203642c468aa6f0039463"}
Jul 14 15:06:14.397635 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:14.397597    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2" event={"ID":"af5e9828-22f9-4bd0-aeb8-45960410ced7","Type":"ContainerStarted","Data":"f9d50d9809d4a248d03496dd25cb4fe8e4db1dded8d69fe7c887cea7f33a079e"}
Jul 14 15:06:14.398078 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:14.397713    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:06:14.424330 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:14.424268    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2" podStartSLOduration=1.839802412 podStartE2EDuration="2.424250052s" totalImagesPullingTime="584.44764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:06:14.423311682 +0000 UTC m=+2152.089498128" watchObservedRunningTime="2026-07-14 15:06:14.424250052 +0000 UTC m=+2152.090436474"
Jul 14 15:06:20.408605 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:20.408577    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:06:23.093157 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:23.093124    2586 scope.go:117] "RemoveContainer" containerID="094caee4350742d170bcd6eb39db54e3b4859029daf043c457143f1297798832"
Jul 14 15:06:23.101721 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:23.101698    2586 scope.go:117] "RemoveContainer" containerID="ba40f2ffed6dfc0098392722811d3e7f08f1cfb036c19070edb00de4f7a22709"
Jul 14 15:06:48.373641 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.373607    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"]
Jul 14 15:06:48.377696 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.377672    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.380934 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.380908    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4226a2\"/\"test-model-4226a2-kserve-self-signed-certs\""
Jul 14 15:06:48.381933 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.381914    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4226a2\"/\"openshift-service-ca.crt\""
Jul 14 15:06:48.383743 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.383719    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4226a2\"/\"kube-root-ca.crt\""
Jul 14 15:06:48.383957 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.383938    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4226a2\"/\"default-dockercfg-8q7cv\""
Jul 14 15:06:48.395678 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.395645    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"]
Jul 14 15:06:48.449619 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.449585    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-model-cache\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.449802 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.449634    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-tmp-dir\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.449802 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.449688    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-home\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.449802 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.449721    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00c1f20-ee19-47a9-a697-e0b85e4b752c-tls-certs\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.449802 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.449737    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-dshm\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.449802 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.449790    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kwjr\" (UniqueName: \"kubernetes.io/projected/e00c1f20-ee19-47a9-a697-e0b85e4b752c-kube-api-access-8kwjr\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.550677 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.550633    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-tmp-dir\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.550874 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.550695    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-home\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.550874 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.550749    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00c1f20-ee19-47a9-a697-e0b85e4b752c-tls-certs\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.550874 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.550774    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-dshm\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.550874 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.550823    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kwjr\" (UniqueName: \"kubernetes.io/projected/e00c1f20-ee19-47a9-a697-e0b85e4b752c-kube-api-access-8kwjr\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.550874 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.550872    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-model-cache\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.551140 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.551082    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-home\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.551140 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.551113    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-tmp-dir\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.551293 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.551273    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-model-cache\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.553220 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.553199    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-dshm\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.553385 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.553369    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00c1f20-ee19-47a9-a697-e0b85e4b752c-tls-certs\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.562696 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.562669    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kwjr\" (UniqueName: \"kubernetes.io/projected/e00c1f20-ee19-47a9-a697-e0b85e4b752c-kube-api-access-8kwjr\") pod \"test-model-4226a2-kserve-799556684c-r8j8f\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") " pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.689584 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.689538    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:48.833236 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:48.833206    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"]
Jul 14 15:06:48.835092 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:06:48.835065    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode00c1f20_ee19_47a9_a697_e0b85e4b752c.slice/crio-e89ac8bb3c0f896550f5eac02ecd361b8a84f375cabf3ad1185f3249f3ba7b39 WatchSource:0}: Error finding container e89ac8bb3c0f896550f5eac02ecd361b8a84f375cabf3ad1185f3249f3ba7b39: Status 404 returned error can't find the container with id e89ac8bb3c0f896550f5eac02ecd361b8a84f375cabf3ad1185f3249f3ba7b39
Jul 14 15:06:49.584262 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:49.584225    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f" event={"ID":"e00c1f20-ee19-47a9-a697-e0b85e4b752c","Type":"ContainerStarted","Data":"7ea7b6ebd9a4763f7bd804e063bc93052dbd09f7286e23162cae101e789c980a"}
Jul 14 15:06:49.584262 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:49.584270    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f" event={"ID":"e00c1f20-ee19-47a9-a697-e0b85e4b752c","Type":"ContainerStarted","Data":"e89ac8bb3c0f896550f5eac02ecd361b8a84f375cabf3ad1185f3249f3ba7b39"}
Jul 14 15:06:49.607249 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:49.607202    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f" podStartSLOduration=1.6071837850000001 podStartE2EDuration="1.607183785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:06:49.603946013 +0000 UTC m=+2187.270132434" watchObservedRunningTime="2026-07-14 15:06:49.607183785 +0000 UTC m=+2187.273370206"
Jul 14 15:06:58.690468 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:58.690399    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:58.690915 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:58.690482    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:58.703378 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:58.703353    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:06:59.652333 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:06:59.652303    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:07:11.546037 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:11.546001    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"]
Jul 14 15:07:22.156005 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.155971    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"]
Jul 14 15:07:22.156555 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.156200    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv" podUID="0252fa82-9c5d-4216-ab03-487d51675632" containerName="maas-api" containerID="cri-o://7ae1702d0514ff3de0c5df13ed870325979636d6463fde2706ad7c0a359be828" gracePeriod=30
Jul 14 15:07:22.357698 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.357638    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv" podUID="0252fa82-9c5d-4216-ab03-487d51675632" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.55:8443/health\": dial tcp 10.133.0.55:8443: connect: connection refused"
Jul 14 15:07:22.605425 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.605387    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:07:22.681404 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.681358    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0252fa82-9c5d-4216-ab03-487d51675632-maas-api-tls\") pod \"0252fa82-9c5d-4216-ab03-487d51675632\" (UID: \"0252fa82-9c5d-4216-ab03-487d51675632\") "
Jul 14 15:07:22.681404 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.681405    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx26c\" (UniqueName: \"kubernetes.io/projected/0252fa82-9c5d-4216-ab03-487d51675632-kube-api-access-mx26c\") pod \"0252fa82-9c5d-4216-ab03-487d51675632\" (UID: \"0252fa82-9c5d-4216-ab03-487d51675632\") "
Jul 14 15:07:22.681687 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.681477    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0252fa82-9c5d-4216-ab03-487d51675632-openshift-service-ca\") pod \"0252fa82-9c5d-4216-ab03-487d51675632\" (UID: \"0252fa82-9c5d-4216-ab03-487d51675632\") "
Jul 14 15:07:22.681899 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.681870    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0252fa82-9c5d-4216-ab03-487d51675632-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0252fa82-9c5d-4216-ab03-487d51675632" (UID: "0252fa82-9c5d-4216-ab03-487d51675632"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:07:22.683775 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.683748    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0252fa82-9c5d-4216-ab03-487d51675632-kube-api-access-mx26c" (OuterVolumeSpecName: "kube-api-access-mx26c") pod "0252fa82-9c5d-4216-ab03-487d51675632" (UID: "0252fa82-9c5d-4216-ab03-487d51675632"). InnerVolumeSpecName "kube-api-access-mx26c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:07:22.683879 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.683800    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0252fa82-9c5d-4216-ab03-487d51675632-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0252fa82-9c5d-4216-ab03-487d51675632" (UID: "0252fa82-9c5d-4216-ab03-487d51675632"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:07:22.761472 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.761407    2586 generic.go:356] "Generic (PLEG): container finished" podID="0252fa82-9c5d-4216-ab03-487d51675632" containerID="7ae1702d0514ff3de0c5df13ed870325979636d6463fde2706ad7c0a359be828" exitCode=0
Jul 14 15:07:22.761472 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.761465    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"
Jul 14 15:07:22.761694 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.761509    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv" event={"ID":"0252fa82-9c5d-4216-ab03-487d51675632","Type":"ContainerDied","Data":"7ae1702d0514ff3de0c5df13ed870325979636d6463fde2706ad7c0a359be828"}
Jul 14 15:07:22.761694 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.761557    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv" event={"ID":"0252fa82-9c5d-4216-ab03-487d51675632","Type":"ContainerDied","Data":"01133f37520cb863c5caec306eb7978d5a15e80f894a62761c592c0ec7e7f597"}
Jul 14 15:07:22.761694 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.761578    2586 scope.go:117] "RemoveContainer" containerID="7ae1702d0514ff3de0c5df13ed870325979636d6463fde2706ad7c0a359be828"
Jul 14 15:07:22.771272 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.771255    2586 scope.go:117] "RemoveContainer" containerID="7ae1702d0514ff3de0c5df13ed870325979636d6463fde2706ad7c0a359be828"
Jul 14 15:07:22.771571 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:07:22.771553    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae1702d0514ff3de0c5df13ed870325979636d6463fde2706ad7c0a359be828\": container with ID starting with 7ae1702d0514ff3de0c5df13ed870325979636d6463fde2706ad7c0a359be828 not found: ID does not exist" containerID="7ae1702d0514ff3de0c5df13ed870325979636d6463fde2706ad7c0a359be828"
Jul 14 15:07:22.771645 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.771580    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae1702d0514ff3de0c5df13ed870325979636d6463fde2706ad7c0a359be828"} err="failed to get container status \"7ae1702d0514ff3de0c5df13ed870325979636d6463fde2706ad7c0a359be828\": rpc error: code = NotFound desc = could not find container \"7ae1702d0514ff3de0c5df13ed870325979636d6463fde2706ad7c0a359be828\": container with ID starting with 7ae1702d0514ff3de0c5df13ed870325979636d6463fde2706ad7c0a359be828 not found: ID does not exist"
Jul 14 15:07:22.782631 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.782604    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0252fa82-9c5d-4216-ab03-487d51675632-maas-api-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:07:22.782631 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.782628    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx26c\" (UniqueName: \"kubernetes.io/projected/0252fa82-9c5d-4216-ab03-487d51675632-kube-api-access-mx26c\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:07:22.782802 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.782639    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0252fa82-9c5d-4216-ab03-487d51675632-openshift-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:07:22.785349 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.785327    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"]
Jul 14 15:07:22.791333 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.791311    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03dca1-6d769cb8b-l7kqv"]
Jul 14 15:07:22.876500 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:22.876471    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0252fa82-9c5d-4216-ab03-487d51675632" path="/var/lib/kubelet/pods/0252fa82-9c5d-4216-ab03-487d51675632/volumes"
Jul 14 15:07:26.567225 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.567170    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f" podUID="e00c1f20-ee19-47a9-a697-e0b85e4b752c" containerName="main" containerID="cri-o://7ea7b6ebd9a4763f7bd804e063bc93052dbd09f7286e23162cae101e789c980a" gracePeriod=45
Jul 14 15:07:26.785584 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.785550    2586 generic.go:356] "Generic (PLEG): container finished" podID="e00c1f20-ee19-47a9-a697-e0b85e4b752c" containerID="7ea7b6ebd9a4763f7bd804e063bc93052dbd09f7286e23162cae101e789c980a" exitCode=0
Jul 14 15:07:26.785753 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.785622    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f" event={"ID":"e00c1f20-ee19-47a9-a697-e0b85e4b752c","Type":"ContainerDied","Data":"7ea7b6ebd9a4763f7bd804e063bc93052dbd09f7286e23162cae101e789c980a"}
Jul 14 15:07:26.825794 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.825734    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:07:26.921406 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.921377    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-model-cache\") pod \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") "
Jul 14 15:07:26.921587 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.921516    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-dshm\") pod \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") "
Jul 14 15:07:26.921587 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.921536    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-home\") pod \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") "
Jul 14 15:07:26.921587 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.921554    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kwjr\" (UniqueName: \"kubernetes.io/projected/e00c1f20-ee19-47a9-a697-e0b85e4b752c-kube-api-access-8kwjr\") pod \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") "
Jul 14 15:07:26.921587 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.921574    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-tmp-dir\") pod \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") "
Jul 14 15:07:26.921764 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.921629    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00c1f20-ee19-47a9-a697-e0b85e4b752c-tls-certs\") pod \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\" (UID: \"e00c1f20-ee19-47a9-a697-e0b85e4b752c\") "
Jul 14 15:07:26.921764 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.921675    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e00c1f20-ee19-47a9-a697-e0b85e4b752c" (UID: "e00c1f20-ee19-47a9-a697-e0b85e4b752c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:07:26.921867 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.921764    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-home" (OuterVolumeSpecName: "home") pod "e00c1f20-ee19-47a9-a697-e0b85e4b752c" (UID: "e00c1f20-ee19-47a9-a697-e0b85e4b752c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:07:26.921925 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.921857    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e00c1f20-ee19-47a9-a697-e0b85e4b752c" (UID: "e00c1f20-ee19-47a9-a697-e0b85e4b752c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:07:26.921994 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.921969    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-model-cache\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:07:26.921994 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.921988    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-home\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:07:26.922063 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.922000    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-tmp-dir\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:07:26.923980 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.923947    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-dshm" (OuterVolumeSpecName: "dshm") pod "e00c1f20-ee19-47a9-a697-e0b85e4b752c" (UID: "e00c1f20-ee19-47a9-a697-e0b85e4b752c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:07:26.923980 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.923971    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00c1f20-ee19-47a9-a697-e0b85e4b752c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e00c1f20-ee19-47a9-a697-e0b85e4b752c" (UID: "e00c1f20-ee19-47a9-a697-e0b85e4b752c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:07:26.924135 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:26.924023    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00c1f20-ee19-47a9-a697-e0b85e4b752c-kube-api-access-8kwjr" (OuterVolumeSpecName: "kube-api-access-8kwjr") pod "e00c1f20-ee19-47a9-a697-e0b85e4b752c" (UID: "e00c1f20-ee19-47a9-a697-e0b85e4b752c"). InnerVolumeSpecName "kube-api-access-8kwjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:07:27.022916 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:27.022877    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00c1f20-ee19-47a9-a697-e0b85e4b752c-dshm\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:07:27.022916 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:27.022909    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kwjr\" (UniqueName: \"kubernetes.io/projected/e00c1f20-ee19-47a9-a697-e0b85e4b752c-kube-api-access-8kwjr\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:07:27.022916 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:27.022918    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00c1f20-ee19-47a9-a697-e0b85e4b752c-tls-certs\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:07:27.791750 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:27.791720    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"
Jul 14 15:07:27.792167 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:27.791764    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f" event={"ID":"e00c1f20-ee19-47a9-a697-e0b85e4b752c","Type":"ContainerDied","Data":"e89ac8bb3c0f896550f5eac02ecd361b8a84f375cabf3ad1185f3249f3ba7b39"}
Jul 14 15:07:27.792167 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:27.791818    2586 scope.go:117] "RemoveContainer" containerID="7ea7b6ebd9a4763f7bd804e063bc93052dbd09f7286e23162cae101e789c980a"
Jul 14 15:07:27.818322 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:27.818293    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"]
Jul 14 15:07:27.820373 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:27.820347    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4226a2/test-model-4226a2-kserve-799556684c-r8j8f"]
Jul 14 15:07:28.876001 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:28.875968    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e00c1f20-ee19-47a9-a697-e0b85e4b752c" path="/var/lib/kubelet/pods/e00c1f20-ee19-47a9-a697-e0b85e4b752c/volumes"
Jul 14 15:07:29.888378 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:07:29.888348    2586 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252fa82_9c5d_4216_ab03_487d51675632.slice\": RecentStats: unable to find data in memory cache]"
Jul 14 15:07:33.113603 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:07:33.113555    2586 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252fa82_9c5d_4216_ab03_487d51675632.slice\": RecentStats: unable to find data in memory cache]"
Jul 14 15:07:37.848267 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:37.848233    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx"]
Jul 14 15:07:37.848758 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:37.848642    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0252fa82-9c5d-4216-ab03-487d51675632" containerName="maas-api"
Jul 14 15:07:37.848758 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:37.848653    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0252fa82-9c5d-4216-ab03-487d51675632" containerName="maas-api"
Jul 14 15:07:37.848758 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:37.848667    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00c1f20-ee19-47a9-a697-e0b85e4b752c" containerName="main"
Jul 14 15:07:37.848758 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:37.848675    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00c1f20-ee19-47a9-a697-e0b85e4b752c" containerName="main"
Jul 14 15:07:37.848980 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:37.848767    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e00c1f20-ee19-47a9-a697-e0b85e4b752c" containerName="main"
Jul 14 15:07:37.848980 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:37.848775    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0252fa82-9c5d-4216-ab03-487d51675632" containerName="maas-api"
Jul 14 15:07:37.855007 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:37.854977    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx"
Jul 14 15:07:37.858689 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:37.858665    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5695q\""
Jul 14 15:07:37.860983 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:37.860953    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx"]
Jul 14 15:07:38.030250 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:38.030211    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ae57172-c7bf-4384-9926-095ff55b3105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx\" (UID: \"0ae57172-c7bf-4384-9926-095ff55b3105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx"
Jul 14 15:07:38.131921 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:38.131832    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ae57172-c7bf-4384-9926-095ff55b3105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx\" (UID: \"0ae57172-c7bf-4384-9926-095ff55b3105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx"
Jul 14 15:07:38.132498 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:38.132479    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ae57172-c7bf-4384-9926-095ff55b3105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx\" (UID: \"0ae57172-c7bf-4384-9926-095ff55b3105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx"
Jul 14 15:07:38.167834 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:38.167802    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx"
Jul 14 15:07:38.311896 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:38.311863    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx"]
Jul 14 15:07:38.313665 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:07:38.313629    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae57172_c7bf_4384_9926_095ff55b3105.slice/crio-ec0448c406e12ef7546292dca9f747e083c2dcfc7632dc322c008f0a3592dec5 WatchSource:0}: Error finding container ec0448c406e12ef7546292dca9f747e083c2dcfc7632dc322c008f0a3592dec5: Status 404 returned error can't find the container with id ec0448c406e12ef7546292dca9f747e083c2dcfc7632dc322c008f0a3592dec5
Jul 14 15:07:38.852954 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:38.852865    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx" event={"ID":"0ae57172-c7bf-4384-9926-095ff55b3105","Type":"ContainerStarted","Data":"0e09f598a9a7f5710ce5f09c7446d12178b981e0cd02b7450894abc0af3251cd"}
Jul 14 15:07:38.852954 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:38.852918    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx" event={"ID":"0ae57172-c7bf-4384-9926-095ff55b3105","Type":"ContainerStarted","Data":"ec0448c406e12ef7546292dca9f747e083c2dcfc7632dc322c008f0a3592dec5"}
Jul 14 15:07:38.870777 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:38.870727    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx" podStartSLOduration=1.87071089 podStartE2EDuration="1.87071089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:07:38.869941621 +0000 UTC m=+2236.536128036" watchObservedRunningTime="2026-07-14 15:07:38.87071089 +0000 UTC m=+2236.536897313"
Jul 14 15:07:39.859467 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:39.859408    2586 generic.go:356] "Generic (PLEG): container finished" podID="0ae57172-c7bf-4384-9926-095ff55b3105" containerID="0e09f598a9a7f5710ce5f09c7446d12178b981e0cd02b7450894abc0af3251cd" exitCode=0
Jul 14 15:07:39.859834 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:39.859510    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx" event={"ID":"0ae57172-c7bf-4384-9926-095ff55b3105","Type":"ContainerDied","Data":"0e09f598a9a7f5710ce5f09c7446d12178b981e0cd02b7450894abc0af3251cd"}
Jul 14 15:07:39.934744 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:07:39.934713    2586 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252fa82_9c5d_4216_ab03_487d51675632.slice\": RecentStats: unable to find data in memory cache]"
Jul 14 15:07:41.001548 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:41.001525    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx"
Jul 14 15:07:41.161162 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:41.161081    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ae57172-c7bf-4384-9926-095ff55b3105-maas-api-service-ca\") pod \"0ae57172-c7bf-4384-9926-095ff55b3105\" (UID: \"0ae57172-c7bf-4384-9926-095ff55b3105\") "
Jul 14 15:07:41.161526 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:41.161503    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae57172-c7bf-4384-9926-095ff55b3105-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ae57172-c7bf-4384-9926-095ff55b3105" (UID: "0ae57172-c7bf-4384-9926-095ff55b3105"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:07:41.262221 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:41.262184    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ae57172-c7bf-4384-9926-095ff55b3105-maas-api-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:07:41.871222 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:41.871192    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx"
Jul 14 15:07:41.871222 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:41.871203    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx" event={"ID":"0ae57172-c7bf-4384-9926-095ff55b3105","Type":"ContainerDied","Data":"ec0448c406e12ef7546292dca9f747e083c2dcfc7632dc322c008f0a3592dec5"}
Jul 14 15:07:41.871450 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:41.871241    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec0448c406e12ef7546292dca9f747e083c2dcfc7632dc322c008f0a3592dec5"
Jul 14 15:07:48.153544 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:07:48.153462    2586 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252fa82_9c5d_4216_ab03_487d51675632.slice\": RecentStats: unable to find data in memory cache]"
Jul 14 15:07:48.372308 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.372272    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"]
Jul 14 15:07:48.372569 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.372542    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2" podUID="af5e9828-22f9-4bd0-aeb8-45960410ced7" containerName="maas-api" containerID="cri-o://f9d50d9809d4a248d03496dd25cb4fe8e4db1dded8d69fe7c887cea7f33a079e" gracePeriod=30
Jul 14 15:07:48.627495 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.627468    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:07:48.734775 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.734740    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af5e9828-22f9-4bd0-aeb8-45960410ced7-maas-api-tls\") pod \"af5e9828-22f9-4bd0-aeb8-45960410ced7\" (UID: \"af5e9828-22f9-4bd0-aeb8-45960410ced7\") "
Jul 14 15:07:48.734976 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.734817    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz55m\" (UniqueName: \"kubernetes.io/projected/af5e9828-22f9-4bd0-aeb8-45960410ced7-kube-api-access-bz55m\") pod \"af5e9828-22f9-4bd0-aeb8-45960410ced7\" (UID: \"af5e9828-22f9-4bd0-aeb8-45960410ced7\") "
Jul 14 15:07:48.734976 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.734919    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af5e9828-22f9-4bd0-aeb8-45960410ced7-openshift-service-ca\") pod \"af5e9828-22f9-4bd0-aeb8-45960410ced7\" (UID: \"af5e9828-22f9-4bd0-aeb8-45960410ced7\") "
Jul 14 15:07:48.735283 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.735258    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5e9828-22f9-4bd0-aeb8-45960410ced7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af5e9828-22f9-4bd0-aeb8-45960410ced7" (UID: "af5e9828-22f9-4bd0-aeb8-45960410ced7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:07:48.737114 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.737085    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af5e9828-22f9-4bd0-aeb8-45960410ced7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af5e9828-22f9-4bd0-aeb8-45960410ced7" (UID: "af5e9828-22f9-4bd0-aeb8-45960410ced7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:07:48.737201 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.737157    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5e9828-22f9-4bd0-aeb8-45960410ced7-kube-api-access-bz55m" (OuterVolumeSpecName: "kube-api-access-bz55m") pod "af5e9828-22f9-4bd0-aeb8-45960410ced7" (UID: "af5e9828-22f9-4bd0-aeb8-45960410ced7"). InnerVolumeSpecName "kube-api-access-bz55m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:07:48.836504 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.836464    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af5e9828-22f9-4bd0-aeb8-45960410ced7-openshift-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:07:48.836504 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.836497    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af5e9828-22f9-4bd0-aeb8-45960410ced7-maas-api-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:07:48.836504 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.836507    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz55m\" (UniqueName: \"kubernetes.io/projected/af5e9828-22f9-4bd0-aeb8-45960410ced7-kube-api-access-bz55m\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:07:48.914611 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.914576    2586 generic.go:356] "Generic (PLEG): container finished" podID="af5e9828-22f9-4bd0-aeb8-45960410ced7" containerID="f9d50d9809d4a248d03496dd25cb4fe8e4db1dded8d69fe7c887cea7f33a079e" exitCode=0
Jul 14 15:07:48.914785 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.914625    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"
Jul 14 15:07:48.914785 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.914656    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2" event={"ID":"af5e9828-22f9-4bd0-aeb8-45960410ced7","Type":"ContainerDied","Data":"f9d50d9809d4a248d03496dd25cb4fe8e4db1dded8d69fe7c887cea7f33a079e"}
Jul 14 15:07:48.914785 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.914708    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2" event={"ID":"af5e9828-22f9-4bd0-aeb8-45960410ced7","Type":"ContainerDied","Data":"0c68471118e1c42ee801257599d83867dabab9ac03b203642c468aa6f0039463"}
Jul 14 15:07:48.914785 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.914729    2586 scope.go:117] "RemoveContainer" containerID="f9d50d9809d4a248d03496dd25cb4fe8e4db1dded8d69fe7c887cea7f33a079e"
Jul 14 15:07:48.923542 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.923527    2586 scope.go:117] "RemoveContainer" containerID="f9d50d9809d4a248d03496dd25cb4fe8e4db1dded8d69fe7c887cea7f33a079e"
Jul 14 15:07:48.923803 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:07:48.923785    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d50d9809d4a248d03496dd25cb4fe8e4db1dded8d69fe7c887cea7f33a079e\": container with ID starting with f9d50d9809d4a248d03496dd25cb4fe8e4db1dded8d69fe7c887cea7f33a079e not found: ID does not exist" containerID="f9d50d9809d4a248d03496dd25cb4fe8e4db1dded8d69fe7c887cea7f33a079e"
Jul 14 15:07:48.923859 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.923811    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d50d9809d4a248d03496dd25cb4fe8e4db1dded8d69fe7c887cea7f33a079e"} err="failed to get container status \"f9d50d9809d4a248d03496dd25cb4fe8e4db1dded8d69fe7c887cea7f33a079e\": rpc error: code = NotFound desc = could not find container \"f9d50d9809d4a248d03496dd25cb4fe8e4db1dded8d69fe7c887cea7f33a079e\": container with ID starting with f9d50d9809d4a248d03496dd25cb4fe8e4db1dded8d69fe7c887cea7f33a079e not found: ID does not exist"
Jul 14 15:07:48.936105 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.936081    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"]
Jul 14 15:07:48.941188 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:48.941166    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4226a2-5bc45f4586-6bfz2"]
Jul 14 15:07:49.944130 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:07:49.944089    2586 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252fa82_9c5d_4216_ab03_487d51675632.slice\": RecentStats: unable to find data in memory cache]"
Jul 14 15:07:50.876678 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:07:50.876644    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af5e9828-22f9-4bd0-aeb8-45960410ced7" path="/var/lib/kubelet/pods/af5e9828-22f9-4bd0-aeb8-45960410ced7/volumes"
Jul 14 15:07:59.989678 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:07:59.989648    2586 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252fa82_9c5d_4216_ab03_487d51675632.slice\": RecentStats: unable to find data in memory cache]"
Jul 14 15:08:03.118229 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:08:03.118193    2586 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252fa82_9c5d_4216_ab03_487d51675632.slice\": RecentStats: unable to find data in memory cache]"
Jul 14 15:08:05.869718 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:08:05.869681    2586 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252fa82_9c5d_4216_ab03_487d51675632.slice\": RecentStats: unable to find data in memory cache]"
Jul 14 15:08:05.869718 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:08:05.869698    2586 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252fa82_9c5d_4216_ab03_487d51675632.slice\": RecentStats: unable to find data in memory cache]"
Jul 14 15:08:08.331150 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.331114    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"]
Jul 14 15:08:08.331724 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.331702    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af5e9828-22f9-4bd0-aeb8-45960410ced7" containerName="maas-api"
Jul 14 15:08:08.331809 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.331728    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="af5e9828-22f9-4bd0-aeb8-45960410ced7" containerName="maas-api"
Jul 14 15:08:08.331809 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.331761    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ae57172-c7bf-4384-9926-095ff55b3105" containerName="revoke-keys"
Jul 14 15:08:08.331809 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.331770    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ae57172-c7bf-4384-9926-095ff55b3105" containerName="revoke-keys"
Jul 14 15:08:08.331964 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.331861    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="af5e9828-22f9-4bd0-aeb8-45960410ced7" containerName="maas-api"
Jul 14 15:08:08.331964 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.331875    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ae57172-c7bf-4384-9926-095ff55b3105" containerName="revoke-keys"
Jul 14 15:08:08.335045 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.335026    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:08.358332 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.358307    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e0a4af-serving-cert\""
Jul 14 15:08:08.397620 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.397591    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"]
Jul 14 15:08:08.419978 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.419947    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579addcd-1b41-4b80-9c03-6155fd67be40-maas-api-tls\") pod \"maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v\" (UID: \"579addcd-1b41-4b80-9c03-6155fd67be40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:08.419978 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.419986    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4qmm\" (UniqueName: \"kubernetes.io/projected/579addcd-1b41-4b80-9c03-6155fd67be40-kube-api-access-z4qmm\") pod \"maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v\" (UID: \"579addcd-1b41-4b80-9c03-6155fd67be40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:08.420216 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.420142    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579addcd-1b41-4b80-9c03-6155fd67be40-openshift-service-ca\") pod \"maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v\" (UID: \"579addcd-1b41-4b80-9c03-6155fd67be40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:08.521036 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.521001    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579addcd-1b41-4b80-9c03-6155fd67be40-maas-api-tls\") pod \"maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v\" (UID: \"579addcd-1b41-4b80-9c03-6155fd67be40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:08.521232 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.521056    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4qmm\" (UniqueName: \"kubernetes.io/projected/579addcd-1b41-4b80-9c03-6155fd67be40-kube-api-access-z4qmm\") pod \"maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v\" (UID: \"579addcd-1b41-4b80-9c03-6155fd67be40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:08.521232 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.521141    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579addcd-1b41-4b80-9c03-6155fd67be40-openshift-service-ca\") pod \"maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v\" (UID: \"579addcd-1b41-4b80-9c03-6155fd67be40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:08.521785 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.521752    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579addcd-1b41-4b80-9c03-6155fd67be40-openshift-service-ca\") pod \"maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v\" (UID: \"579addcd-1b41-4b80-9c03-6155fd67be40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:08.523597 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.523578    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579addcd-1b41-4b80-9c03-6155fd67be40-maas-api-tls\") pod \"maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v\" (UID: \"579addcd-1b41-4b80-9c03-6155fd67be40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:08.531326 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.531299    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4qmm\" (UniqueName: \"kubernetes.io/projected/579addcd-1b41-4b80-9c03-6155fd67be40-kube-api-access-z4qmm\") pod \"maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v\" (UID: \"579addcd-1b41-4b80-9c03-6155fd67be40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:08.645074 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.644996    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:08.805334 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.805307    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"]
Jul 14 15:08:08.807471 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:08:08.807439    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579addcd_1b41_4b80_9c03_6155fd67be40.slice/crio-c13396fc9ff6c680c8326a30f48fc8775d16198f7812d20f4b841ed446a558a3 WatchSource:0}: Error finding container c13396fc9ff6c680c8326a30f48fc8775d16198f7812d20f4b841ed446a558a3: Status 404 returned error can't find the container with id c13396fc9ff6c680c8326a30f48fc8775d16198f7812d20f4b841ed446a558a3
Jul 14 15:08:08.808838 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:08.808821    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 15:08:09.025713 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:09.025679    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v" event={"ID":"579addcd-1b41-4b80-9c03-6155fd67be40","Type":"ContainerStarted","Data":"c13396fc9ff6c680c8326a30f48fc8775d16198f7812d20f4b841ed446a558a3"}
Jul 14 15:08:09.999666 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:08:09.999627    2586 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252fa82_9c5d_4216_ab03_487d51675632.slice\": RecentStats: unable to find data in memory cache]"
Jul 14 15:08:10.032493 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:10.032461    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v" event={"ID":"579addcd-1b41-4b80-9c03-6155fd67be40","Type":"ContainerStarted","Data":"22e35f1b506fcdc6aca9b7c7268fbfb42337f220a5b2c1ac4c274a30c451f217"}
Jul 14 15:08:10.032671 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:10.032517    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:10.054088 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:10.054038    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v" podStartSLOduration=1.6367535800000002 podStartE2EDuration="2.054022659s" totalImagesPullingTime="417.269079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:08:10.051494457 +0000 UTC m=+2267.717680891" watchObservedRunningTime="2026-07-14 15:08:10.054022659 +0000 UTC m=+2267.720209079"
Jul 14 15:08:16.043493 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:16.043463    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:17.712230 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:17.712185    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"]
Jul 14 15:08:17.715932 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:17.715912    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:08:17.721648 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:17.721628    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-40b1bd-serving-cert\""
Jul 14 15:08:17.746573 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:17.746547    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"]
Jul 14 15:08:17.810520 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:17.809716    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/056c78a1-f89c-4b66-8164-1b33f193f4e2-maas-api-tls\") pod \"maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx\" (UID: \"056c78a1-f89c-4b66-8164-1b33f193f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:08:17.810520 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:17.809781    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79gkt\" (UniqueName: \"kubernetes.io/projected/056c78a1-f89c-4b66-8164-1b33f193f4e2-kube-api-access-79gkt\") pod \"maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx\" (UID: \"056c78a1-f89c-4b66-8164-1b33f193f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:08:17.810520 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:17.809836    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/056c78a1-f89c-4b66-8164-1b33f193f4e2-openshift-service-ca\") pod \"maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx\" (UID: \"056c78a1-f89c-4b66-8164-1b33f193f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:08:17.911228 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:17.911191    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gkt\" (UniqueName: \"kubernetes.io/projected/056c78a1-f89c-4b66-8164-1b33f193f4e2-kube-api-access-79gkt\") pod \"maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx\" (UID: \"056c78a1-f89c-4b66-8164-1b33f193f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:08:17.911373 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:17.911245    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/056c78a1-f89c-4b66-8164-1b33f193f4e2-openshift-service-ca\") pod \"maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx\" (UID: \"056c78a1-f89c-4b66-8164-1b33f193f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:08:17.911453 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:17.911367    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/056c78a1-f89c-4b66-8164-1b33f193f4e2-maas-api-tls\") pod \"maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx\" (UID: \"056c78a1-f89c-4b66-8164-1b33f193f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:08:17.912054 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:17.912028    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/056c78a1-f89c-4b66-8164-1b33f193f4e2-openshift-service-ca\") pod \"maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx\" (UID: \"056c78a1-f89c-4b66-8164-1b33f193f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:08:17.913866 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:17.913847    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/056c78a1-f89c-4b66-8164-1b33f193f4e2-maas-api-tls\") pod \"maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx\" (UID: \"056c78a1-f89c-4b66-8164-1b33f193f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:08:17.928866 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:17.928838    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gkt\" (UniqueName: \"kubernetes.io/projected/056c78a1-f89c-4b66-8164-1b33f193f4e2-kube-api-access-79gkt\") pod \"maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx\" (UID: \"056c78a1-f89c-4b66-8164-1b33f193f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:08:18.026236 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:18.026155    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:08:18.160024 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:08:18.159984    2586 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252fa82_9c5d_4216_ab03_487d51675632.slice\": RecentStats: unable to find data in memory cache]"
Jul 14 15:08:18.176273 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:18.176247    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"]
Jul 14 15:08:18.178246 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:08:18.178216    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056c78a1_f89c_4b66_8164_1b33f193f4e2.slice/crio-d44ead0f0ca347de8a87bb3babb59f09429d13f8761266e5e3f9efffa6991fcd WatchSource:0}: Error finding container d44ead0f0ca347de8a87bb3babb59f09429d13f8761266e5e3f9efffa6991fcd: Status 404 returned error can't find the container with id d44ead0f0ca347de8a87bb3babb59f09429d13f8761266e5e3f9efffa6991fcd
Jul 14 15:08:19.082231 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:19.082197    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx" event={"ID":"056c78a1-f89c-4b66-8164-1b33f193f4e2","Type":"ContainerStarted","Data":"c705ba17c73832eb2584098f894329d4bf271ff1c11011078531fcd6dbd8a86c"}
Jul 14 15:08:19.082688 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:19.082249    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx" event={"ID":"056c78a1-f89c-4b66-8164-1b33f193f4e2","Type":"ContainerStarted","Data":"d44ead0f0ca347de8a87bb3babb59f09429d13f8761266e5e3f9efffa6991fcd"}
Jul 14 15:08:19.082688 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:19.082437    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:08:19.102234 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:19.102184    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx" podStartSLOduration=1.716784552 podStartE2EDuration="2.102168478s" totalImagesPullingTime="385.383926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:08:19.100520363 +0000 UTC m=+2276.766706782" watchObservedRunningTime="2026-07-14 15:08:19.102168478 +0000 UTC m=+2276.768354936"
Jul 14 15:08:20.009654 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:08:20.009616    2586 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252fa82_9c5d_4216_ab03_487d51675632.slice\": RecentStats: unable to find data in memory cache]"
Jul 14 15:08:25.094851 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:25.094822    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:08:40.884984 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:40.884947    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"]
Jul 14 15:08:40.885462 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:40.885173    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v" podUID="579addcd-1b41-4b80-9c03-6155fd67be40" containerName="maas-api" containerID="cri-o://22e35f1b506fcdc6aca9b7c7268fbfb42337f220a5b2c1ac4c274a30c451f217" gracePeriod=30
Jul 14 15:08:41.038508 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.038462    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v" podUID="579addcd-1b41-4b80-9c03-6155fd67be40" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": dial tcp 10.133.0.59:8443: connect: connection refused"
Jul 14 15:08:41.142388 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.142329    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:41.203659 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.203627    2586 generic.go:356] "Generic (PLEG): container finished" podID="579addcd-1b41-4b80-9c03-6155fd67be40" containerID="22e35f1b506fcdc6aca9b7c7268fbfb42337f220a5b2c1ac4c274a30c451f217" exitCode=0
Jul 14 15:08:41.203867 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.203680    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"
Jul 14 15:08:41.203867 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.203712    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v" event={"ID":"579addcd-1b41-4b80-9c03-6155fd67be40","Type":"ContainerDied","Data":"22e35f1b506fcdc6aca9b7c7268fbfb42337f220a5b2c1ac4c274a30c451f217"}
Jul 14 15:08:41.203867 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.203764    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v" event={"ID":"579addcd-1b41-4b80-9c03-6155fd67be40","Type":"ContainerDied","Data":"c13396fc9ff6c680c8326a30f48fc8775d16198f7812d20f4b841ed446a558a3"}
Jul 14 15:08:41.203867 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.203787    2586 scope.go:117] "RemoveContainer" containerID="22e35f1b506fcdc6aca9b7c7268fbfb42337f220a5b2c1ac4c274a30c451f217"
Jul 14 15:08:41.212555 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.212537    2586 scope.go:117] "RemoveContainer" containerID="22e35f1b506fcdc6aca9b7c7268fbfb42337f220a5b2c1ac4c274a30c451f217"
Jul 14 15:08:41.212848 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:08:41.212826    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e35f1b506fcdc6aca9b7c7268fbfb42337f220a5b2c1ac4c274a30c451f217\": container with ID starting with 22e35f1b506fcdc6aca9b7c7268fbfb42337f220a5b2c1ac4c274a30c451f217 not found: ID does not exist" containerID="22e35f1b506fcdc6aca9b7c7268fbfb42337f220a5b2c1ac4c274a30c451f217"
Jul 14 15:08:41.212913 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.212858    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e35f1b506fcdc6aca9b7c7268fbfb42337f220a5b2c1ac4c274a30c451f217"} err="failed to get container status \"22e35f1b506fcdc6aca9b7c7268fbfb42337f220a5b2c1ac4c274a30c451f217\": rpc error: code = NotFound desc = could not find container \"22e35f1b506fcdc6aca9b7c7268fbfb42337f220a5b2c1ac4c274a30c451f217\": container with ID starting with 22e35f1b506fcdc6aca9b7c7268fbfb42337f220a5b2c1ac4c274a30c451f217 not found: ID does not exist"
Jul 14 15:08:41.232315 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.232289    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579addcd-1b41-4b80-9c03-6155fd67be40-maas-api-tls\") pod \"579addcd-1b41-4b80-9c03-6155fd67be40\" (UID: \"579addcd-1b41-4b80-9c03-6155fd67be40\") "
Jul 14 15:08:41.232477 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.232356    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579addcd-1b41-4b80-9c03-6155fd67be40-openshift-service-ca\") pod \"579addcd-1b41-4b80-9c03-6155fd67be40\" (UID: \"579addcd-1b41-4b80-9c03-6155fd67be40\") "
Jul 14 15:08:41.232477 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.232394    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4qmm\" (UniqueName: \"kubernetes.io/projected/579addcd-1b41-4b80-9c03-6155fd67be40-kube-api-access-z4qmm\") pod \"579addcd-1b41-4b80-9c03-6155fd67be40\" (UID: \"579addcd-1b41-4b80-9c03-6155fd67be40\") "
Jul 14 15:08:41.232702 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.232682    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/579addcd-1b41-4b80-9c03-6155fd67be40-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "579addcd-1b41-4b80-9c03-6155fd67be40" (UID: "579addcd-1b41-4b80-9c03-6155fd67be40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:08:41.232804 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.232782    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579addcd-1b41-4b80-9c03-6155fd67be40-openshift-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:08:41.234650 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.234628    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/579addcd-1b41-4b80-9c03-6155fd67be40-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "579addcd-1b41-4b80-9c03-6155fd67be40" (UID: "579addcd-1b41-4b80-9c03-6155fd67be40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:08:41.234706 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.234672    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579addcd-1b41-4b80-9c03-6155fd67be40-kube-api-access-z4qmm" (OuterVolumeSpecName: "kube-api-access-z4qmm") pod "579addcd-1b41-4b80-9c03-6155fd67be40" (UID: "579addcd-1b41-4b80-9c03-6155fd67be40"). InnerVolumeSpecName "kube-api-access-z4qmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:08:41.333432 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.333384    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4qmm\" (UniqueName: \"kubernetes.io/projected/579addcd-1b41-4b80-9c03-6155fd67be40-kube-api-access-z4qmm\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:08:41.333432 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.333435    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579addcd-1b41-4b80-9c03-6155fd67be40-maas-api-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:08:41.527694 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.527661    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"]
Jul 14 15:08:41.532347 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:41.532317    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0a4af-7f475766c8-ghd4v"]
Jul 14 15:08:42.882924 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:42.882888    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="579addcd-1b41-4b80-9c03-6155fd67be40" path="/var/lib/kubelet/pods/579addcd-1b41-4b80-9c03-6155fd67be40/volumes"
Jul 14 15:08:56.819997 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:56.819958    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl"]
Jul 14 15:08:56.820483 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:56.820397    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="579addcd-1b41-4b80-9c03-6155fd67be40" containerName="maas-api"
Jul 14 15:08:56.820483 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:56.820408    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="579addcd-1b41-4b80-9c03-6155fd67be40" containerName="maas-api"
Jul 14 15:08:56.820563 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:56.820486    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="579addcd-1b41-4b80-9c03-6155fd67be40" containerName="maas-api"
Jul 14 15:08:56.825142 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:56.825122    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl"
Jul 14 15:08:56.828010 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:56.827988    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5695q\""
Jul 14 15:08:56.833479 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:56.833455    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl"]
Jul 14 15:08:56.979948 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:56.979904    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e242f140-3f9b-4446-8b0a-e7dbfc455d20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl\" (UID: \"e242f140-3f9b-4446-8b0a-e7dbfc455d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl"
Jul 14 15:08:57.081885 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:57.081800    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e242f140-3f9b-4446-8b0a-e7dbfc455d20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl\" (UID: \"e242f140-3f9b-4446-8b0a-e7dbfc455d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl"
Jul 14 15:08:57.082446 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:57.082400    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e242f140-3f9b-4446-8b0a-e7dbfc455d20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl\" (UID: \"e242f140-3f9b-4446-8b0a-e7dbfc455d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl"
Jul 14 15:08:57.138228 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:57.138195    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl"
Jul 14 15:08:57.292771 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:57.292750    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl"]
Jul 14 15:08:57.295208 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:08:57.295174    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode242f140_3f9b_4446_8b0a_e7dbfc455d20.slice/crio-16055cab40b8dabbd1d5717d8ebc95c9eb0075706cdaae609041105c7b0e2e39 WatchSource:0}: Error finding container 16055cab40b8dabbd1d5717d8ebc95c9eb0075706cdaae609041105c7b0e2e39: Status 404 returned error can't find the container with id 16055cab40b8dabbd1d5717d8ebc95c9eb0075706cdaae609041105c7b0e2e39
Jul 14 15:08:58.295749 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:58.295658    2586 generic.go:356] "Generic (PLEG): container finished" podID="e242f140-3f9b-4446-8b0a-e7dbfc455d20" containerID="24b61df45e6ea90fd87058593437dbecd7fd129e505eb3daf7c5d62993dd44c4" exitCode=0
Jul 14 15:08:58.295749 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:58.295694    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl" event={"ID":"e242f140-3f9b-4446-8b0a-e7dbfc455d20","Type":"ContainerDied","Data":"24b61df45e6ea90fd87058593437dbecd7fd129e505eb3daf7c5d62993dd44c4"}
Jul 14 15:08:58.295749 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:58.295732    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl" event={"ID":"e242f140-3f9b-4446-8b0a-e7dbfc455d20","Type":"ContainerStarted","Data":"16055cab40b8dabbd1d5717d8ebc95c9eb0075706cdaae609041105c7b0e2e39"}
Jul 14 15:08:59.436306 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:59.436279    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl"
Jul 14 15:08:59.607767 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:59.607670    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e242f140-3f9b-4446-8b0a-e7dbfc455d20-maas-api-service-ca\") pod \"e242f140-3f9b-4446-8b0a-e7dbfc455d20\" (UID: \"e242f140-3f9b-4446-8b0a-e7dbfc455d20\") "
Jul 14 15:08:59.608023 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:59.608002    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e242f140-3f9b-4446-8b0a-e7dbfc455d20-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e242f140-3f9b-4446-8b0a-e7dbfc455d20" (UID: "e242f140-3f9b-4446-8b0a-e7dbfc455d20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:08:59.709159 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:08:59.709124    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e242f140-3f9b-4446-8b0a-e7dbfc455d20-maas-api-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:09:00.306941 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:00.306902    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl" event={"ID":"e242f140-3f9b-4446-8b0a-e7dbfc455d20","Type":"ContainerDied","Data":"16055cab40b8dabbd1d5717d8ebc95c9eb0075706cdaae609041105c7b0e2e39"}
Jul 14 15:09:00.306941 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:00.306937    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl"
Jul 14 15:09:00.307164 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:00.306984    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16055cab40b8dabbd1d5717d8ebc95c9eb0075706cdaae609041105c7b0e2e39"
Jul 14 15:09:07.459903 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:07.459864    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"]
Jul 14 15:09:07.460308 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:07.460153    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx" podUID="056c78a1-f89c-4b66-8164-1b33f193f4e2" containerName="maas-api" containerID="cri-o://c705ba17c73832eb2584098f894329d4bf271ff1c11011078531fcd6dbd8a86c" gracePeriod=30
Jul 14 15:09:07.713189 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:07.713124    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:09:07.773722 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:07.773685    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79gkt\" (UniqueName: \"kubernetes.io/projected/056c78a1-f89c-4b66-8164-1b33f193f4e2-kube-api-access-79gkt\") pod \"056c78a1-f89c-4b66-8164-1b33f193f4e2\" (UID: \"056c78a1-f89c-4b66-8164-1b33f193f4e2\") "
Jul 14 15:09:07.773886 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:07.773730    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/056c78a1-f89c-4b66-8164-1b33f193f4e2-maas-api-tls\") pod \"056c78a1-f89c-4b66-8164-1b33f193f4e2\" (UID: \"056c78a1-f89c-4b66-8164-1b33f193f4e2\") "
Jul 14 15:09:07.773886 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:07.773766    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/056c78a1-f89c-4b66-8164-1b33f193f4e2-openshift-service-ca\") pod \"056c78a1-f89c-4b66-8164-1b33f193f4e2\" (UID: \"056c78a1-f89c-4b66-8164-1b33f193f4e2\") "
Jul 14 15:09:07.774197 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:07.774172    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056c78a1-f89c-4b66-8164-1b33f193f4e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "056c78a1-f89c-4b66-8164-1b33f193f4e2" (UID: "056c78a1-f89c-4b66-8164-1b33f193f4e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:09:07.775862 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:07.775839    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056c78a1-f89c-4b66-8164-1b33f193f4e2-kube-api-access-79gkt" (OuterVolumeSpecName: "kube-api-access-79gkt") pod "056c78a1-f89c-4b66-8164-1b33f193f4e2" (UID: "056c78a1-f89c-4b66-8164-1b33f193f4e2"). InnerVolumeSpecName "kube-api-access-79gkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:09:07.775955 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:07.775882    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056c78a1-f89c-4b66-8164-1b33f193f4e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "056c78a1-f89c-4b66-8164-1b33f193f4e2" (UID: "056c78a1-f89c-4b66-8164-1b33f193f4e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:09:07.874624 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:07.874587    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79gkt\" (UniqueName: \"kubernetes.io/projected/056c78a1-f89c-4b66-8164-1b33f193f4e2-kube-api-access-79gkt\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:09:07.874624 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:07.874617    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/056c78a1-f89c-4b66-8164-1b33f193f4e2-maas-api-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:09:07.874624 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:07.874627    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/056c78a1-f89c-4b66-8164-1b33f193f4e2-openshift-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:09:08.349102 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:08.349069    2586 generic.go:356] "Generic (PLEG): container finished" podID="056c78a1-f89c-4b66-8164-1b33f193f4e2" containerID="c705ba17c73832eb2584098f894329d4bf271ff1c11011078531fcd6dbd8a86c" exitCode=0
Jul 14 15:09:08.349281 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:08.349109    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"
Jul 14 15:09:08.349281 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:08.349126    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx" event={"ID":"056c78a1-f89c-4b66-8164-1b33f193f4e2","Type":"ContainerDied","Data":"c705ba17c73832eb2584098f894329d4bf271ff1c11011078531fcd6dbd8a86c"}
Jul 14 15:09:08.349281 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:08.349180    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx" event={"ID":"056c78a1-f89c-4b66-8164-1b33f193f4e2","Type":"ContainerDied","Data":"d44ead0f0ca347de8a87bb3babb59f09429d13f8761266e5e3f9efffa6991fcd"}
Jul 14 15:09:08.349281 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:08.349205    2586 scope.go:117] "RemoveContainer" containerID="c705ba17c73832eb2584098f894329d4bf271ff1c11011078531fcd6dbd8a86c"
Jul 14 15:09:08.358705 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:08.358678    2586 scope.go:117] "RemoveContainer" containerID="c705ba17c73832eb2584098f894329d4bf271ff1c11011078531fcd6dbd8a86c"
Jul 14 15:09:08.358968 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:09:08.358952    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c705ba17c73832eb2584098f894329d4bf271ff1c11011078531fcd6dbd8a86c\": container with ID starting with c705ba17c73832eb2584098f894329d4bf271ff1c11011078531fcd6dbd8a86c not found: ID does not exist" containerID="c705ba17c73832eb2584098f894329d4bf271ff1c11011078531fcd6dbd8a86c"
Jul 14 15:09:08.359026 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:08.358977    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c705ba17c73832eb2584098f894329d4bf271ff1c11011078531fcd6dbd8a86c"} err="failed to get container status \"c705ba17c73832eb2584098f894329d4bf271ff1c11011078531fcd6dbd8a86c\": rpc error: code = NotFound desc = could not find container \"c705ba17c73832eb2584098f894329d4bf271ff1c11011078531fcd6dbd8a86c\": container with ID starting with c705ba17c73832eb2584098f894329d4bf271ff1c11011078531fcd6dbd8a86c not found: ID does not exist"
Jul 14 15:09:08.376611 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:08.376571    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"]
Jul 14 15:09:08.381871 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:08.381844    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40b1bd-5cb7886b7c-65whx"]
Jul 14 15:09:08.877478 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:08.877407    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="056c78a1-f89c-4b66-8164-1b33f193f4e2" path="/var/lib/kubelet/pods/056c78a1-f89c-4b66-8164-1b33f193f4e2/volumes"
Jul 14 15:09:26.703320 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.703286    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"]
Jul 14 15:09:26.703829 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.703811    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056c78a1-f89c-4b66-8164-1b33f193f4e2" containerName="maas-api"
Jul 14 15:09:26.703887 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.703833    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="056c78a1-f89c-4b66-8164-1b33f193f4e2" containerName="maas-api"
Jul 14 15:09:26.703887 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.703850    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e242f140-3f9b-4446-8b0a-e7dbfc455d20" containerName="revoke-keys"
Jul 14 15:09:26.703887 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.703857    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e242f140-3f9b-4446-8b0a-e7dbfc455d20" containerName="revoke-keys"
Jul 14 15:09:26.704037 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.703965    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="056c78a1-f89c-4b66-8164-1b33f193f4e2" containerName="maas-api"
Jul 14 15:09:26.704037 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.703978    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e242f140-3f9b-4446-8b0a-e7dbfc455d20" containerName="revoke-keys"
Jul 14 15:09:26.708871 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.708850    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:09:26.711866 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.711840    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-ebb1a4-serving-cert\""
Jul 14 15:09:26.721522 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.721499    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"]
Jul 14 15:09:26.853224 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.853187    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/383bc700-f214-41ce-94d6-41b624636719-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv\" (UID: \"383bc700-f214-41ce-94d6-41b624636719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:09:26.853401 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.853230    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/383bc700-f214-41ce-94d6-41b624636719-maas-api-tls\") pod \"maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv\" (UID: \"383bc700-f214-41ce-94d6-41b624636719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:09:26.853401 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.853264    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c42n7\" (UniqueName: \"kubernetes.io/projected/383bc700-f214-41ce-94d6-41b624636719-kube-api-access-c42n7\") pod \"maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv\" (UID: \"383bc700-f214-41ce-94d6-41b624636719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:09:26.954360 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.954276    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/383bc700-f214-41ce-94d6-41b624636719-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv\" (UID: \"383bc700-f214-41ce-94d6-41b624636719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:09:26.954360 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.954319    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/383bc700-f214-41ce-94d6-41b624636719-maas-api-tls\") pod \"maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv\" (UID: \"383bc700-f214-41ce-94d6-41b624636719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:09:26.954360 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.954344    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c42n7\" (UniqueName: \"kubernetes.io/projected/383bc700-f214-41ce-94d6-41b624636719-kube-api-access-c42n7\") pod \"maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv\" (UID: \"383bc700-f214-41ce-94d6-41b624636719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:09:26.954688 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:09:26.954464    2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-serving-cert: secret "maas-api-e2e-shared-a-ebb1a4-serving-cert" not found
Jul 14 15:09:26.954688 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:09:26.954528    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/383bc700-f214-41ce-94d6-41b624636719-maas-api-tls podName:383bc700-f214-41ce-94d6-41b624636719 nodeName:}" failed. No retries permitted until 2026-07-14 15:09:27.454512354 +0000 UTC m=+2345.120698753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/383bc700-f214-41ce-94d6-41b624636719-maas-api-tls") pod "maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv" (UID: "383bc700-f214-41ce-94d6-41b624636719") : secret "maas-api-e2e-shared-a-ebb1a4-serving-cert" not found
Jul 14 15:09:26.954933 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.954916    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/383bc700-f214-41ce-94d6-41b624636719-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv\" (UID: \"383bc700-f214-41ce-94d6-41b624636719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:09:26.964834 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:26.964806    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c42n7\" (UniqueName: \"kubernetes.io/projected/383bc700-f214-41ce-94d6-41b624636719-kube-api-access-c42n7\") pod \"maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv\" (UID: \"383bc700-f214-41ce-94d6-41b624636719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:09:27.459277 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:27.459247    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/383bc700-f214-41ce-94d6-41b624636719-maas-api-tls\") pod \"maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv\" (UID: \"383bc700-f214-41ce-94d6-41b624636719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:09:27.461950 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:27.461920    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/383bc700-f214-41ce-94d6-41b624636719-maas-api-tls\") pod \"maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv\" (UID: \"383bc700-f214-41ce-94d6-41b624636719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:09:27.620236 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:27.620201    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:09:27.759802 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:27.759779    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"]
Jul 14 15:09:27.762081 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:09:27.762052    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383bc700_f214_41ce_94d6_41b624636719.slice/crio-79db80ddffa9c6c5f0d1d5d683936955c68dcda653023b7837cb450b93703ef2 WatchSource:0}: Error finding container 79db80ddffa9c6c5f0d1d5d683936955c68dcda653023b7837cb450b93703ef2: Status 404 returned error can't find the container with id 79db80ddffa9c6c5f0d1d5d683936955c68dcda653023b7837cb450b93703ef2
Jul 14 15:09:28.455555 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:28.455524    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv" event={"ID":"383bc700-f214-41ce-94d6-41b624636719","Type":"ContainerStarted","Data":"2ca9da949db93d634ef8de721e9b4db2d628456827dfb8c7819a5f71d9fadc75"}
Jul 14 15:09:28.455759 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:28.455574    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv" event={"ID":"383bc700-f214-41ce-94d6-41b624636719","Type":"ContainerStarted","Data":"79db80ddffa9c6c5f0d1d5d683936955c68dcda653023b7837cb450b93703ef2"}
Jul 14 15:09:28.455759 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:28.455627    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:09:28.478790 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:28.478552    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv" podStartSLOduration=1.9803877349999999 podStartE2EDuration="2.478531815s" totalImagesPullingTime="498.14408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:09:28.477298689 +0000 UTC m=+2346.143485112" watchObservedRunningTime="2026-07-14 15:09:28.478531815 +0000 UTC m=+2346.144718238"
Jul 14 15:09:34.466479 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:34.466446    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:09:34.700502 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:34.700466    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"]
Jul 14 15:09:34.704329 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:34.704311    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:09:34.707521 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:34.707499    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-747ac7-serving-cert\""
Jul 14 15:09:34.718766 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:34.718713    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"]
Jul 14 15:09:34.830030 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:34.829996    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e436be0-6e51-4042-bb5f-8d77b15710b5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z\" (UID: \"5e436be0-6e51-4042-bb5f-8d77b15710b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:09:34.830030 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:34.830030    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbbps\" (UniqueName: \"kubernetes.io/projected/5e436be0-6e51-4042-bb5f-8d77b15710b5-kube-api-access-pbbps\") pod \"maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z\" (UID: \"5e436be0-6e51-4042-bb5f-8d77b15710b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:09:34.830239 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:34.830129    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e436be0-6e51-4042-bb5f-8d77b15710b5-maas-api-tls\") pod \"maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z\" (UID: \"5e436be0-6e51-4042-bb5f-8d77b15710b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:09:34.931432 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:34.931374    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e436be0-6e51-4042-bb5f-8d77b15710b5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z\" (UID: \"5e436be0-6e51-4042-bb5f-8d77b15710b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:09:34.931432 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:34.931426    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbbps\" (UniqueName: \"kubernetes.io/projected/5e436be0-6e51-4042-bb5f-8d77b15710b5-kube-api-access-pbbps\") pod \"maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z\" (UID: \"5e436be0-6e51-4042-bb5f-8d77b15710b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:09:34.931655 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:34.931474    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e436be0-6e51-4042-bb5f-8d77b15710b5-maas-api-tls\") pod \"maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z\" (UID: \"5e436be0-6e51-4042-bb5f-8d77b15710b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:09:34.932202 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:34.932119    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e436be0-6e51-4042-bb5f-8d77b15710b5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z\" (UID: \"5e436be0-6e51-4042-bb5f-8d77b15710b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:09:34.934092 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:34.934068    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e436be0-6e51-4042-bb5f-8d77b15710b5-maas-api-tls\") pod \"maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z\" (UID: \"5e436be0-6e51-4042-bb5f-8d77b15710b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:09:34.941128 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:34.941107    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbbps\" (UniqueName: \"kubernetes.io/projected/5e436be0-6e51-4042-bb5f-8d77b15710b5-kube-api-access-pbbps\") pod \"maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z\" (UID: \"5e436be0-6e51-4042-bb5f-8d77b15710b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:09:35.014609 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:35.014528    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:09:35.158178 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:35.158147    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"]
Jul 14 15:09:35.162604 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:09:35.162574    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e436be0_6e51_4042_bb5f_8d77b15710b5.slice/crio-bf74d8a304134956fb22c33e49d22b1bf8f611d88e687550bffba3fcc3c35a0d WatchSource:0}: Error finding container bf74d8a304134956fb22c33e49d22b1bf8f611d88e687550bffba3fcc3c35a0d: Status 404 returned error can't find the container with id bf74d8a304134956fb22c33e49d22b1bf8f611d88e687550bffba3fcc3c35a0d
Jul 14 15:09:35.492973 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:35.492938    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z" event={"ID":"5e436be0-6e51-4042-bb5f-8d77b15710b5","Type":"ContainerStarted","Data":"bf74d8a304134956fb22c33e49d22b1bf8f611d88e687550bffba3fcc3c35a0d"}
Jul 14 15:09:36.499709 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:36.499672    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z" event={"ID":"5e436be0-6e51-4042-bb5f-8d77b15710b5","Type":"ContainerStarted","Data":"9e719d1208f5718e5f69ec17963fd2003fbc19a03951a1bf7ede66ae4b29f466"}
Jul 14 15:09:36.500179 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:36.499786    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:09:36.520891 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:36.520841    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z" podStartSLOduration=1.967821363 podStartE2EDuration="2.520823063s" totalImagesPullingTime="553.0017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:09:36.519118452 +0000 UTC m=+2354.185304872" watchObservedRunningTime="2026-07-14 15:09:36.520823063 +0000 UTC m=+2354.187009483"
Jul 14 15:09:42.511078 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:09:42.511051    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:10:18.562839 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.562806    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"]
Jul 14 15:10:18.566550 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.566527    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.574039 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.574020    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-747ac7\"/\"kube-root-ca.crt\""
Jul 14 15:10:18.575023 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.574993    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-747ac7\"/\"auth-test-model-747ac7-kserve-self-signed-certs\""
Jul 14 15:10:18.575347 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.575327    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-747ac7\"/\"openshift-service-ca.crt\""
Jul 14 15:10:18.575503 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.575392    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-747ac7\"/\"default-dockercfg-qkp67\""
Jul 14 15:10:18.583761 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.583733    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"]
Jul 14 15:10:18.643778 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.643745    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-model-cache\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.643778 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.643779    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-dshm\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.644037 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.643805    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68n78\" (UniqueName: \"kubernetes.io/projected/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-kube-api-access-68n78\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.644037 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.643931    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-tmp-dir\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.644144 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.644041    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-tls-certs\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.644144 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.644077    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-home\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.744927 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.744886    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-tmp-dir\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.745098 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.745002    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-tls-certs\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.745098 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.745030    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-home\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.745098 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.745082    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-model-cache\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.745229 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.745110    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-dshm\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.745229 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.745147    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68n78\" (UniqueName: \"kubernetes.io/projected/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-kube-api-access-68n78\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.745362 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.745326    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-tmp-dir\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.745451 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.745366    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-home\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.745531 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.745452    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-model-cache\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.747523 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.747495    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-dshm\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.747722 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.747702    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-tls-certs\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.757175 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.757151    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68n78\" (UniqueName: \"kubernetes.io/projected/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-kube-api-access-68n78\") pod \"auth-test-model-747ac7-kserve-68cf694bb7-bzz2z\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") " pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:18.875805 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:18.875720    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:19.015519 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:19.015492    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"]
Jul 14 15:10:19.017124 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:10:19.017099    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf39e2c99_f5b8_4466_8dd8_a189c0c7efc0.slice/crio-25ec5aa3d57908ed7697a06388b4bf98e5f75cf9fe4c2e6a999653a1a6d9f6eb WatchSource:0}: Error finding container 25ec5aa3d57908ed7697a06388b4bf98e5f75cf9fe4c2e6a999653a1a6d9f6eb: Status 404 returned error can't find the container with id 25ec5aa3d57908ed7697a06388b4bf98e5f75cf9fe4c2e6a999653a1a6d9f6eb
Jul 14 15:10:19.725774 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:19.725735    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z" event={"ID":"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0","Type":"ContainerStarted","Data":"2519c03fbe61603f9cb2ae8934e2300c214a0b5dd885e8a56b482dad762e0476"}
Jul 14 15:10:19.725774 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:19.725776    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z" event={"ID":"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0","Type":"ContainerStarted","Data":"25ec5aa3d57908ed7697a06388b4bf98e5f75cf9fe4c2e6a999653a1a6d9f6eb"}
Jul 14 15:10:19.749548 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:19.749502    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z" podStartSLOduration=1.749486718 podStartE2EDuration="1.749486718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:10:19.748384032 +0000 UTC m=+2397.414570464" watchObservedRunningTime="2026-07-14 15:10:19.749486718 +0000 UTC m=+2397.415673139"
Jul 14 15:10:23.010018 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:23.009992    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 15:10:23.014607 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:23.014584    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 15:10:28.877076 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:28.877045    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:28.877509 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:28.877098    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:28.890448 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:28.890399    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:10:29.789436 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:10:29.789394    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:12:17.976152 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:17.976069    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"]
Jul 14 15:12:32.998491 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:32.998408    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z" podUID="f39e2c99-f5b8-4466-8dd8-a189c0c7efc0" containerName="main" containerID="cri-o://2519c03fbe61603f9cb2ae8934e2300c214a0b5dd885e8a56b482dad762e0476" gracePeriod=45
Jul 14 15:12:33.255671 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.255605    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:12:33.329898 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.329870    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-tmp-dir\") pod \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") "
Jul 14 15:12:33.330071 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.329943    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-dshm\") pod \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") "
Jul 14 15:12:33.330071 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.329984    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-home\") pod \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") "
Jul 14 15:12:33.330071 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.330006    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68n78\" (UniqueName: \"kubernetes.io/projected/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-kube-api-access-68n78\") pod \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") "
Jul 14 15:12:33.330071 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.330033    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-model-cache\") pod \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") "
Jul 14 15:12:33.330293 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.330086    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-tls-certs\") pod \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\" (UID: \"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0\") "
Jul 14 15:12:33.330293 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.330191    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f39e2c99-f5b8-4466-8dd8-a189c0c7efc0" (UID: "f39e2c99-f5b8-4466-8dd8-a189c0c7efc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:12:33.330293 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.330262    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-home" (OuterVolumeSpecName: "home") pod "f39e2c99-f5b8-4466-8dd8-a189c0c7efc0" (UID: "f39e2c99-f5b8-4466-8dd8-a189c0c7efc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:12:33.330466 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.330343    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-model-cache" (OuterVolumeSpecName: "model-cache") pod "f39e2c99-f5b8-4466-8dd8-a189c0c7efc0" (UID: "f39e2c99-f5b8-4466-8dd8-a189c0c7efc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:12:33.330466 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.330397    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-home\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:12:33.330466 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.330409    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-tmp-dir\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:12:33.332367 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.332349    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-dshm" (OuterVolumeSpecName: "dshm") pod "f39e2c99-f5b8-4466-8dd8-a189c0c7efc0" (UID: "f39e2c99-f5b8-4466-8dd8-a189c0c7efc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:12:33.332485 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.332437    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-kube-api-access-68n78" (OuterVolumeSpecName: "kube-api-access-68n78") pod "f39e2c99-f5b8-4466-8dd8-a189c0c7efc0" (UID: "f39e2c99-f5b8-4466-8dd8-a189c0c7efc0"). InnerVolumeSpecName "kube-api-access-68n78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:12:33.332485 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.332472    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f39e2c99-f5b8-4466-8dd8-a189c0c7efc0" (UID: "f39e2c99-f5b8-4466-8dd8-a189c0c7efc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:12:33.431710 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.431670    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-dshm\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:12:33.431710 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.431700    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68n78\" (UniqueName: \"kubernetes.io/projected/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-kube-api-access-68n78\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:12:33.431710 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.431715    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-model-cache\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:12:33.432025 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.431725    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0-tls-certs\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:12:33.437129 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.437101    2586 generic.go:356] "Generic (PLEG): container finished" podID="f39e2c99-f5b8-4466-8dd8-a189c0c7efc0" containerID="2519c03fbe61603f9cb2ae8934e2300c214a0b5dd885e8a56b482dad762e0476" exitCode=0
Jul 14 15:12:33.437276 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.437143    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"
Jul 14 15:12:33.437276 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.437154    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z" event={"ID":"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0","Type":"ContainerDied","Data":"2519c03fbe61603f9cb2ae8934e2300c214a0b5dd885e8a56b482dad762e0476"}
Jul 14 15:12:33.437276 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.437186    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z" event={"ID":"f39e2c99-f5b8-4466-8dd8-a189c0c7efc0","Type":"ContainerDied","Data":"25ec5aa3d57908ed7697a06388b4bf98e5f75cf9fe4c2e6a999653a1a6d9f6eb"}
Jul 14 15:12:33.437276 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.437201    2586 scope.go:117] "RemoveContainer" containerID="2519c03fbe61603f9cb2ae8934e2300c214a0b5dd885e8a56b482dad762e0476"
Jul 14 15:12:33.447289 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.447270    2586 scope.go:117] "RemoveContainer" containerID="2519c03fbe61603f9cb2ae8934e2300c214a0b5dd885e8a56b482dad762e0476"
Jul 14 15:12:33.447556 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:12:33.447536    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2519c03fbe61603f9cb2ae8934e2300c214a0b5dd885e8a56b482dad762e0476\": container with ID starting with 2519c03fbe61603f9cb2ae8934e2300c214a0b5dd885e8a56b482dad762e0476 not found: ID does not exist" containerID="2519c03fbe61603f9cb2ae8934e2300c214a0b5dd885e8a56b482dad762e0476"
Jul 14 15:12:33.447604 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.447564    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2519c03fbe61603f9cb2ae8934e2300c214a0b5dd885e8a56b482dad762e0476"} err="failed to get container status \"2519c03fbe61603f9cb2ae8934e2300c214a0b5dd885e8a56b482dad762e0476\": rpc error: code = NotFound desc = could not find container \"2519c03fbe61603f9cb2ae8934e2300c214a0b5dd885e8a56b482dad762e0476\": container with ID starting with 2519c03fbe61603f9cb2ae8934e2300c214a0b5dd885e8a56b482dad762e0476 not found: ID does not exist"
Jul 14 15:12:33.460808 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.460781    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"]
Jul 14 15:12:33.465258 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:33.465237    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-747ac7/auth-test-model-747ac7-kserve-68cf694bb7-bzz2z"]
Jul 14 15:12:34.877092 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:34.877057    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f39e2c99-f5b8-4466-8dd8-a189c0c7efc0" path="/var/lib/kubelet/pods/f39e2c99-f5b8-4466-8dd8-a189c0c7efc0/volumes"
Jul 14 15:12:45.061578 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.061545    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"]
Jul 14 15:12:45.062113 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.061969    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f39e2c99-f5b8-4466-8dd8-a189c0c7efc0" containerName="main"
Jul 14 15:12:45.062113 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.061985    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f39e2c99-f5b8-4466-8dd8-a189c0c7efc0" containerName="main"
Jul 14 15:12:45.062113 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.062040    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f39e2c99-f5b8-4466-8dd8-a189c0c7efc0" containerName="main"
Jul 14 15:12:45.066447 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.066428    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.069380 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.069360    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-747ac7\"/\"kube-root-ca.crt\""
Jul 14 15:12:45.070674 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.070657    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-747ac7\"/\"default-dockercfg-qkp67\""
Jul 14 15:12:45.070674 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.070670    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-747ac7\"/\"openshift-service-ca.crt\""
Jul 14 15:12:45.070769 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.070703    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-747ac7\"/\"sub-test-model-747ac7-kserve-self-signed-certs\""
Jul 14 15:12:45.085388 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.085352    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"]
Jul 14 15:12:45.143586 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.143542    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-home\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.143767 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.143611    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-dshm\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.143767 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.143633    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-tmp-dir\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.143767 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.143665    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-tls-certs\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.143767 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.143699    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ct4s\" (UniqueName: \"kubernetes.io/projected/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-kube-api-access-7ct4s\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.143767 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.143718    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-model-cache\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.245152 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.245108    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-home\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.245327 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.245178    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-dshm\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.245327 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.245197    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-tmp-dir\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.245327 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.245213    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-tls-certs\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.245327 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.245249    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ct4s\" (UniqueName: \"kubernetes.io/projected/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-kube-api-access-7ct4s\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.245327 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.245269    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-model-cache\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.245703 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.245677    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-home\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.245780 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.245700    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-tmp-dir\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.245780 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.245755    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-model-cache\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.247682 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.247662    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-dshm\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.248072 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.248051    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-tls-certs\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.254195 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.254166    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ct4s\" (UniqueName: \"kubernetes.io/projected/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-kube-api-access-7ct4s\") pod \"sub-test-model-747ac7-kserve-6445b559df-hjzq8\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") " pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.380736 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.380591    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:45.527592 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:45.527561    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"]
Jul 14 15:12:45.529477 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:12:45.529448    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2430b8d_e0d0_4c5a_a65f_b220373ddafe.slice/crio-3e6b9ad43bf2b2af3322f15c062dd1d937288797c723a774a5f8565b98214033 WatchSource:0}: Error finding container 3e6b9ad43bf2b2af3322f15c062dd1d937288797c723a774a5f8565b98214033: Status 404 returned error can't find the container with id 3e6b9ad43bf2b2af3322f15c062dd1d937288797c723a774a5f8565b98214033
Jul 14 15:12:46.509102 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:46.509066    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8" event={"ID":"f2430b8d-e0d0-4c5a-a65f-b220373ddafe","Type":"ContainerStarted","Data":"b8be7c8103a1b250d7327ce36c82b7faf11cdbfcdb3440ce52873ba3c977fcd5"}
Jul 14 15:12:46.509600 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:46.509116    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8" event={"ID":"f2430b8d-e0d0-4c5a-a65f-b220373ddafe","Type":"ContainerStarted","Data":"3e6b9ad43bf2b2af3322f15c062dd1d937288797c723a774a5f8565b98214033"}
Jul 14 15:12:46.528651 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:46.528599    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8" podStartSLOduration=1.5285833210000002 podStartE2EDuration="1.528583321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:12:46.526842432 +0000 UTC m=+2544.193028847" watchObservedRunningTime="2026-07-14 15:12:46.528583321 +0000 UTC m=+2544.194769742"
Jul 14 15:12:55.381211 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:55.381166    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:55.381841 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:55.381256    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:55.394121 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:55.394096    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:12:55.568119 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:12:55.568093    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:13:24.645185 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:24.645151    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"]
Jul 14 15:13:39.665809 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:39.665763    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8" podUID="f2430b8d-e0d0-4c5a-a65f-b220373ddafe" containerName="main" containerID="cri-o://b8be7c8103a1b250d7327ce36c82b7faf11cdbfcdb3440ce52873ba3c977fcd5" gracePeriod=45
Jul 14 15:13:39.799372 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:39.799327    2586 generic.go:356] "Generic (PLEG): container finished" podID="f2430b8d-e0d0-4c5a-a65f-b220373ddafe" containerID="b8be7c8103a1b250d7327ce36c82b7faf11cdbfcdb3440ce52873ba3c977fcd5" exitCode=0
Jul 14 15:13:39.799558 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:39.799430    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8" event={"ID":"f2430b8d-e0d0-4c5a-a65f-b220373ddafe","Type":"ContainerDied","Data":"b8be7c8103a1b250d7327ce36c82b7faf11cdbfcdb3440ce52873ba3c977fcd5"}
Jul 14 15:13:39.925121 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:39.925099    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:13:40.076232 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.076199    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-tmp-dir\") pod \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") "
Jul 14 15:13:40.076438 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.076242    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-tls-certs\") pod \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") "
Jul 14 15:13:40.076438 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.076293    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-model-cache\") pod \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") "
Jul 14 15:13:40.076438 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.076368    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ct4s\" (UniqueName: \"kubernetes.io/projected/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-kube-api-access-7ct4s\") pod \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") "
Jul 14 15:13:40.076560 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.076439    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-home\") pod \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") "
Jul 14 15:13:40.076560 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.076479    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-dshm\") pod \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\" (UID: \"f2430b8d-e0d0-4c5a-a65f-b220373ddafe\") "
Jul 14 15:13:40.076560 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.076531    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2430b8d-e0d0-4c5a-a65f-b220373ddafe" (UID: "f2430b8d-e0d0-4c5a-a65f-b220373ddafe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:13:40.076670 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.076565    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2430b8d-e0d0-4c5a-a65f-b220373ddafe" (UID: "f2430b8d-e0d0-4c5a-a65f-b220373ddafe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:13:40.076854 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.076821    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-tmp-dir\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:13:40.076854 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.076855    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-model-cache\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:13:40.077258 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.076925    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-home" (OuterVolumeSpecName: "home") pod "f2430b8d-e0d0-4c5a-a65f-b220373ddafe" (UID: "f2430b8d-e0d0-4c5a-a65f-b220373ddafe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:13:40.078972 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.078946    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-dshm" (OuterVolumeSpecName: "dshm") pod "f2430b8d-e0d0-4c5a-a65f-b220373ddafe" (UID: "f2430b8d-e0d0-4c5a-a65f-b220373ddafe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 15:13:40.079188 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.079158    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-kube-api-access-7ct4s" (OuterVolumeSpecName: "kube-api-access-7ct4s") pod "f2430b8d-e0d0-4c5a-a65f-b220373ddafe" (UID: "f2430b8d-e0d0-4c5a-a65f-b220373ddafe"). InnerVolumeSpecName "kube-api-access-7ct4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:13:40.079188 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.079170    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2430b8d-e0d0-4c5a-a65f-b220373ddafe" (UID: "f2430b8d-e0d0-4c5a-a65f-b220373ddafe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:13:40.178053 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.177952    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ct4s\" (UniqueName: \"kubernetes.io/projected/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-kube-api-access-7ct4s\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:13:40.178053 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.177983    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-home\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:13:40.178053 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.177993    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-dshm\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:13:40.178053 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.178001    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2430b8d-e0d0-4c5a-a65f-b220373ddafe-tls-certs\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:13:40.807490 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.807462    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"
Jul 14 15:13:40.807490 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.807476    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8" event={"ID":"f2430b8d-e0d0-4c5a-a65f-b220373ddafe","Type":"ContainerDied","Data":"3e6b9ad43bf2b2af3322f15c062dd1d937288797c723a774a5f8565b98214033"}
Jul 14 15:13:40.807979 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.807531    2586 scope.go:117] "RemoveContainer" containerID="b8be7c8103a1b250d7327ce36c82b7faf11cdbfcdb3440ce52873ba3c977fcd5"
Jul 14 15:13:40.831193 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.831160    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"]
Jul 14 15:13:40.836204 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:13:40.836175    2586 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2430b8d_e0d0_4c5a_a65f_b220373ddafe.slice/crio-3e6b9ad43bf2b2af3322f15c062dd1d937288797c723a774a5f8565b98214033\": RecentStats: unable to find data in memory cache]"
Jul 14 15:13:40.836960 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.836940    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-747ac7/sub-test-model-747ac7-kserve-6445b559df-hjzq8"]
Jul 14 15:13:40.876983 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:13:40.876951    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2430b8d-e0d0-4c5a-a65f-b220373ddafe" path="/var/lib/kubelet/pods/f2430b8d-e0d0-4c5a-a65f-b220373ddafe/volumes"
Jul 14 15:14:42.119842 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:42.119812    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7489-140672880217648"]
Jul 14 15:14:42.120288 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:42.120209    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2430b8d-e0d0-4c5a-a65f-b220373ddafe" containerName="main"
Jul 14 15:14:42.120288 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:42.120222    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2430b8d-e0d0-4c5a-a65f-b220373ddafe" containerName="main"
Jul 14 15:14:42.120360 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:42.120294    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2430b8d-e0d0-4c5a-a65f-b220373ddafe" containerName="main"
Jul 14 15:14:42.123540 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:42.123522    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672880217648"
Jul 14 15:14:42.126650 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:42.126592    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-84wgl\""
Jul 14 15:14:42.129398 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:42.129375    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7489-140672880217648"]
Jul 14 15:14:42.158830 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:42.158799    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46dsk\" (UniqueName: \"kubernetes.io/projected/6d42481c-1448-4368-bce2-6820dbc3b4eb-kube-api-access-46dsk\") pod \"test-curl-7489-140672880217648\" (UID: \"6d42481c-1448-4368-bce2-6820dbc3b4eb\") " pod="openshift-ingress/test-curl-7489-140672880217648"
Jul 14 15:14:42.259837 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:42.259799    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46dsk\" (UniqueName: \"kubernetes.io/projected/6d42481c-1448-4368-bce2-6820dbc3b4eb-kube-api-access-46dsk\") pod \"test-curl-7489-140672880217648\" (UID: \"6d42481c-1448-4368-bce2-6820dbc3b4eb\") " pod="openshift-ingress/test-curl-7489-140672880217648"
Jul 14 15:14:42.269645 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:42.269619    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46dsk\" (UniqueName: \"kubernetes.io/projected/6d42481c-1448-4368-bce2-6820dbc3b4eb-kube-api-access-46dsk\") pod \"test-curl-7489-140672880217648\" (UID: \"6d42481c-1448-4368-bce2-6820dbc3b4eb\") " pod="openshift-ingress/test-curl-7489-140672880217648"
Jul 14 15:14:42.433745 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:42.433716    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672880217648"
Jul 14 15:14:42.572461 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:42.572430    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7489-140672880217648"]
Jul 14 15:14:42.574480 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:14:42.574441    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d42481c_1448_4368_bce2_6820dbc3b4eb.slice/crio-3a4028fb7c5e20216625d1a70a931a0a0b3f4374e5f1be45d53543d73385a11f WatchSource:0}: Error finding container 3a4028fb7c5e20216625d1a70a931a0a0b3f4374e5f1be45d53543d73385a11f: Status 404 returned error can't find the container with id 3a4028fb7c5e20216625d1a70a931a0a0b3f4374e5f1be45d53543d73385a11f
Jul 14 15:14:42.575755 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:42.575739    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 15:14:43.147104 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:43.147045    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7489-140672880217648" event={"ID":"6d42481c-1448-4368-bce2-6820dbc3b4eb","Type":"ContainerStarted","Data":"3a4028fb7c5e20216625d1a70a931a0a0b3f4374e5f1be45d53543d73385a11f"}
Jul 14 15:14:44.153255 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:44.153222    2586 generic.go:356] "Generic (PLEG): container finished" podID="6d42481c-1448-4368-bce2-6820dbc3b4eb" containerID="b718fe16e34ede19ba88549b307f4fb3c8c0170b1b326ac5397c606509b3bd82" exitCode=0
Jul 14 15:14:44.153659 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:44.153287    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7489-140672880217648" event={"ID":"6d42481c-1448-4368-bce2-6820dbc3b4eb","Type":"ContainerDied","Data":"b718fe16e34ede19ba88549b307f4fb3c8c0170b1b326ac5397c606509b3bd82"}
Jul 14 15:14:45.313747 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.313723    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672880217648"
Jul 14 15:14:45.363579 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.363548    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7489-140672880217648_6d42481c-1448-4368-bce2-6820dbc3b4eb/test-curl-7489-140672880217648/0.log"
Jul 14 15:14:45.392661 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.392632    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46dsk\" (UniqueName: \"kubernetes.io/projected/6d42481c-1448-4368-bce2-6820dbc3b4eb-kube-api-access-46dsk\") pod \"6d42481c-1448-4368-bce2-6820dbc3b4eb\" (UID: \"6d42481c-1448-4368-bce2-6820dbc3b4eb\") "
Jul 14 15:14:45.394960 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.394922    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d42481c-1448-4368-bce2-6820dbc3b4eb-kube-api-access-46dsk" (OuterVolumeSpecName: "kube-api-access-46dsk") pod "6d42481c-1448-4368-bce2-6820dbc3b4eb" (UID: "6d42481c-1448-4368-bce2-6820dbc3b4eb"). InnerVolumeSpecName "kube-api-access-46dsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:14:45.401785 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.401764    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7489-140672880217648"]
Jul 14 15:14:45.405214 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.405194    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7489-140672880217648"]
Jul 14 15:14:45.494094 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.494053    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46dsk\" (UniqueName: \"kubernetes.io/projected/6d42481c-1448-4368-bce2-6820dbc3b4eb-kube-api-access-46dsk\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:14:45.612380 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.612345    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7489-140672879851184"]
Jul 14 15:14:45.612795 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.612780    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d42481c-1448-4368-bce2-6820dbc3b4eb" containerName="test-curl-7489-140672880217648"
Jul 14 15:14:45.612841 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.612797    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d42481c-1448-4368-bce2-6820dbc3b4eb" containerName="test-curl-7489-140672880217648"
Jul 14 15:14:45.612879 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.612873    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d42481c-1448-4368-bce2-6820dbc3b4eb" containerName="test-curl-7489-140672880217648"
Jul 14 15:14:45.616088 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.616073    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879851184"
Jul 14 15:14:45.624189 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.624165    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7489-140672879851184"]
Jul 14 15:14:45.695474 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.695403    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kplqv\" (UniqueName: \"kubernetes.io/projected/378c7996-3b48-4487-87e7-94b2620355ed-kube-api-access-kplqv\") pod \"test-curl-7489-140672879851184\" (UID: \"378c7996-3b48-4487-87e7-94b2620355ed\") " pod="openshift-ingress/test-curl-7489-140672879851184"
Jul 14 15:14:45.796177 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.796084    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kplqv\" (UniqueName: \"kubernetes.io/projected/378c7996-3b48-4487-87e7-94b2620355ed-kube-api-access-kplqv\") pod \"test-curl-7489-140672879851184\" (UID: \"378c7996-3b48-4487-87e7-94b2620355ed\") " pod="openshift-ingress/test-curl-7489-140672879851184"
Jul 14 15:14:45.807562 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.807534    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kplqv\" (UniqueName: \"kubernetes.io/projected/378c7996-3b48-4487-87e7-94b2620355ed-kube-api-access-kplqv\") pod \"test-curl-7489-140672879851184\" (UID: \"378c7996-3b48-4487-87e7-94b2620355ed\") " pod="openshift-ingress/test-curl-7489-140672879851184"
Jul 14 15:14:45.926429 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:45.926389    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879851184"
Jul 14 15:14:46.064759 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:46.064732    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7489-140672879851184"]
Jul 14 15:14:46.066049 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:14:46.066017    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod378c7996_3b48_4487_87e7_94b2620355ed.slice/crio-4ce30ad472586fc38cf8580645dbe5c37f9adb6b45094aa1ce92c8869e26e753 WatchSource:0}: Error finding container 4ce30ad472586fc38cf8580645dbe5c37f9adb6b45094aa1ce92c8869e26e753: Status 404 returned error can't find the container with id 4ce30ad472586fc38cf8580645dbe5c37f9adb6b45094aa1ce92c8869e26e753
Jul 14 15:14:46.164654 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:46.164615    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7489-140672879851184" event={"ID":"378c7996-3b48-4487-87e7-94b2620355ed","Type":"ContainerStarted","Data":"4ce30ad472586fc38cf8580645dbe5c37f9adb6b45094aa1ce92c8869e26e753"}
Jul 14 15:14:46.165847 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:46.165830    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a4028fb7c5e20216625d1a70a931a0a0b3f4374e5f1be45d53543d73385a11f"
Jul 14 15:14:46.165946 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:46.165864    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672880217648"
Jul 14 15:14:46.176340 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:14:46.176301    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7489-140672880217648\" is forbidden: User \"system:node:ip-10-0-141-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-209.ec2.internal' and this object" podUID="6d42481c-1448-4368-bce2-6820dbc3b4eb" pod="openshift-ingress/test-curl-7489-140672880217648"
Jul 14 15:14:46.876772 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:46.876740    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d42481c-1448-4368-bce2-6820dbc3b4eb" path="/var/lib/kubelet/pods/6d42481c-1448-4368-bce2-6820dbc3b4eb/volumes"
Jul 14 15:14:47.172809 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:47.172725    2586 generic.go:356] "Generic (PLEG): container finished" podID="378c7996-3b48-4487-87e7-94b2620355ed" containerID="0ff5205d3729e93e3e0f62382d5bdc3c6aadf1dcfdaea7a645fdc1cac214387f" exitCode=0
Jul 14 15:14:47.172809 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:47.172767    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7489-140672879851184" event={"ID":"378c7996-3b48-4487-87e7-94b2620355ed","Type":"ContainerDied","Data":"0ff5205d3729e93e3e0f62382d5bdc3c6aadf1dcfdaea7a645fdc1cac214387f"}
Jul 14 15:14:48.322034 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.322005    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879851184"
Jul 14 15:14:48.341871 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.341835    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7489-140672879851184_378c7996-3b48-4487-87e7-94b2620355ed/test-curl-7489-140672879851184/0.log"
Jul 14 15:14:48.379599 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.379566    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7489-140672879851184"]
Jul 14 15:14:48.385426 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.385386    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7489-140672879851184"]
Jul 14 15:14:48.417293 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.417260    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kplqv\" (UniqueName: \"kubernetes.io/projected/378c7996-3b48-4487-87e7-94b2620355ed-kube-api-access-kplqv\") pod \"378c7996-3b48-4487-87e7-94b2620355ed\" (UID: \"378c7996-3b48-4487-87e7-94b2620355ed\") "
Jul 14 15:14:48.419579 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.419554    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378c7996-3b48-4487-87e7-94b2620355ed-kube-api-access-kplqv" (OuterVolumeSpecName: "kube-api-access-kplqv") pod "378c7996-3b48-4487-87e7-94b2620355ed" (UID: "378c7996-3b48-4487-87e7-94b2620355ed"). InnerVolumeSpecName "kube-api-access-kplqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:14:48.518975 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.518890    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kplqv\" (UniqueName: \"kubernetes.io/projected/378c7996-3b48-4487-87e7-94b2620355ed-kube-api-access-kplqv\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:14:48.651846 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.651804    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7489-140672879943728"]
Jul 14 15:14:48.652336 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.652315    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="378c7996-3b48-4487-87e7-94b2620355ed" containerName="test-curl-7489-140672879851184"
Jul 14 15:14:48.652449 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.652339    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="378c7996-3b48-4487-87e7-94b2620355ed" containerName="test-curl-7489-140672879851184"
Jul 14 15:14:48.652516 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.652450    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="378c7996-3b48-4487-87e7-94b2620355ed" containerName="test-curl-7489-140672879851184"
Jul 14 15:14:48.655447 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.655408    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879943728"
Jul 14 15:14:48.666236 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.666216    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7489-140672879943728"]
Jul 14 15:14:48.720597 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.720566    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqgxt\" (UniqueName: \"kubernetes.io/projected/cdd426ac-fa61-4742-b376-fd48a30adda0-kube-api-access-zqgxt\") pod \"test-curl-7489-140672879943728\" (UID: \"cdd426ac-fa61-4742-b376-fd48a30adda0\") " pod="openshift-ingress/test-curl-7489-140672879943728"
Jul 14 15:14:48.821376 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.821295    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqgxt\" (UniqueName: \"kubernetes.io/projected/cdd426ac-fa61-4742-b376-fd48a30adda0-kube-api-access-zqgxt\") pod \"test-curl-7489-140672879943728\" (UID: \"cdd426ac-fa61-4742-b376-fd48a30adda0\") " pod="openshift-ingress/test-curl-7489-140672879943728"
Jul 14 15:14:48.832840 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.832815    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqgxt\" (UniqueName: \"kubernetes.io/projected/cdd426ac-fa61-4742-b376-fd48a30adda0-kube-api-access-zqgxt\") pod \"test-curl-7489-140672879943728\" (UID: \"cdd426ac-fa61-4742-b376-fd48a30adda0\") " pod="openshift-ingress/test-curl-7489-140672879943728"
Jul 14 15:14:48.876528 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.876498    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="378c7996-3b48-4487-87e7-94b2620355ed" path="/var/lib/kubelet/pods/378c7996-3b48-4487-87e7-94b2620355ed/volumes"
Jul 14 15:14:48.964345 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:48.964311    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879943728"
Jul 14 15:14:49.120213 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:49.120184    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7489-140672879943728"]
Jul 14 15:14:49.121675 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:14:49.121649    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdd426ac_fa61_4742_b376_fd48a30adda0.slice/crio-02ffd1fc66a315531c97899ff171e7d9370568692493b9413f6adbafe4182477 WatchSource:0}: Error finding container 02ffd1fc66a315531c97899ff171e7d9370568692493b9413f6adbafe4182477: Status 404 returned error can't find the container with id 02ffd1fc66a315531c97899ff171e7d9370568692493b9413f6adbafe4182477
Jul 14 15:14:49.185554 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:49.185522    2586 scope.go:117] "RemoveContainer" containerID="0ff5205d3729e93e3e0f62382d5bdc3c6aadf1dcfdaea7a645fdc1cac214387f"
Jul 14 15:14:49.185554 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:49.185552    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879851184"
Jul 14 15:14:49.186997 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:49.186737    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7489-140672879943728" event={"ID":"cdd426ac-fa61-4742-b376-fd48a30adda0","Type":"ContainerStarted","Data":"02ffd1fc66a315531c97899ff171e7d9370568692493b9413f6adbafe4182477"}
Jul 14 15:14:50.194530 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:50.194496    2586 generic.go:356] "Generic (PLEG): container finished" podID="cdd426ac-fa61-4742-b376-fd48a30adda0" containerID="f68d4d2fb396d67ab86af36c94283c5957ed094c44febaa467f86d19ae2df300" exitCode=0
Jul 14 15:14:50.194950 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:50.194597    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7489-140672879943728" event={"ID":"cdd426ac-fa61-4742-b376-fd48a30adda0","Type":"ContainerDied","Data":"f68d4d2fb396d67ab86af36c94283c5957ed094c44febaa467f86d19ae2df300"}
Jul 14 15:14:51.344044 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.344014    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879943728"
Jul 14 15:14:51.370566 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.370536    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7489-140672879943728_cdd426ac-fa61-4742-b376-fd48a30adda0/test-curl-7489-140672879943728/0.log"
Jul 14 15:14:51.406621 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.406590    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7489-140672879943728"]
Jul 14 15:14:51.413517 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.413494    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7489-140672879943728"]
Jul 14 15:14:51.447868 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.447828    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqgxt\" (UniqueName: \"kubernetes.io/projected/cdd426ac-fa61-4742-b376-fd48a30adda0-kube-api-access-zqgxt\") pod \"cdd426ac-fa61-4742-b376-fd48a30adda0\" (UID: \"cdd426ac-fa61-4742-b376-fd48a30adda0\") "
Jul 14 15:14:51.450011 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.449984    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd426ac-fa61-4742-b376-fd48a30adda0-kube-api-access-zqgxt" (OuterVolumeSpecName: "kube-api-access-zqgxt") pod "cdd426ac-fa61-4742-b376-fd48a30adda0" (UID: "cdd426ac-fa61-4742-b376-fd48a30adda0"). InnerVolumeSpecName "kube-api-access-zqgxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:14:51.549076 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.549039    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqgxt\" (UniqueName: \"kubernetes.io/projected/cdd426ac-fa61-4742-b376-fd48a30adda0-kube-api-access-zqgxt\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:14:51.614451 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.614400    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7489-140672879943728"]
Jul 14 15:14:51.614831 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.614818    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdd426ac-fa61-4742-b376-fd48a30adda0" containerName="test-curl-7489-140672879943728"
Jul 14 15:14:51.614831 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.614832    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdd426ac-fa61-4742-b376-fd48a30adda0" containerName="test-curl-7489-140672879943728"
Jul 14 15:14:51.614929 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.614891    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdd426ac-fa61-4742-b376-fd48a30adda0" containerName="test-curl-7489-140672879943728"
Jul 14 15:14:51.618051 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.618035    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879943728"
Jul 14 15:14:51.624247 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.624219    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7489-140672879943728"]
Jul 14 15:14:51.750439 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.750299    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtmf9\" (UniqueName: \"kubernetes.io/projected/479d44a6-26ec-412f-ab83-8d558457f65d-kube-api-access-mtmf9\") pod \"test-curl-7489-140672879943728\" (UID: \"479d44a6-26ec-412f-ab83-8d558457f65d\") " pod="openshift-ingress/test-curl-7489-140672879943728"
Jul 14 15:14:51.851667 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.851636    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtmf9\" (UniqueName: \"kubernetes.io/projected/479d44a6-26ec-412f-ab83-8d558457f65d-kube-api-access-mtmf9\") pod \"test-curl-7489-140672879943728\" (UID: \"479d44a6-26ec-412f-ab83-8d558457f65d\") " pod="openshift-ingress/test-curl-7489-140672879943728"
Jul 14 15:14:51.862018 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.861990    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtmf9\" (UniqueName: \"kubernetes.io/projected/479d44a6-26ec-412f-ab83-8d558457f65d-kube-api-access-mtmf9\") pod \"test-curl-7489-140672879943728\" (UID: \"479d44a6-26ec-412f-ab83-8d558457f65d\") " pod="openshift-ingress/test-curl-7489-140672879943728"
Jul 14 15:14:51.928439 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:51.928389    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879943728"
Jul 14 15:14:52.070137 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:52.070109    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7489-140672879943728"]
Jul 14 15:14:52.071524 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:14:52.071495    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod479d44a6_26ec_412f_ab83_8d558457f65d.slice/crio-ef852aabbdefab1a0b3a5bea2653b218abbc34a884547ff35ca2ed7781826999 WatchSource:0}: Error finding container ef852aabbdefab1a0b3a5bea2653b218abbc34a884547ff35ca2ed7781826999: Status 404 returned error can't find the container with id ef852aabbdefab1a0b3a5bea2653b218abbc34a884547ff35ca2ed7781826999
Jul 14 15:14:52.206361 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:52.206321    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7489-140672879943728" event={"ID":"479d44a6-26ec-412f-ab83-8d558457f65d","Type":"ContainerStarted","Data":"ef852aabbdefab1a0b3a5bea2653b218abbc34a884547ff35ca2ed7781826999"}
Jul 14 15:14:52.207559 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:52.207530    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02ffd1fc66a315531c97899ff171e7d9370568692493b9413f6adbafe4182477"
Jul 14 15:14:52.207559 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:52.207543    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879943728"
Jul 14 15:14:52.219847 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:52.219817    2586 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7489-140672879943728" oldPodUID="cdd426ac-fa61-4742-b376-fd48a30adda0" podUID="479d44a6-26ec-412f-ab83-8d558457f65d"
Jul 14 15:14:52.876955 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:52.876923    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdd426ac-fa61-4742-b376-fd48a30adda0" path="/var/lib/kubelet/pods/cdd426ac-fa61-4742-b376-fd48a30adda0/volumes"
Jul 14 15:14:53.214755 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:53.214716    2586 generic.go:356] "Generic (PLEG): container finished" podID="479d44a6-26ec-412f-ab83-8d558457f65d" containerID="cd2fbb1e622bbc8e8204d0f52a4e6614e06b5b9845d56c549cae1720073d7742" exitCode=0
Jul 14 15:14:53.214955 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:53.214797    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7489-140672879943728" event={"ID":"479d44a6-26ec-412f-ab83-8d558457f65d","Type":"ContainerDied","Data":"cd2fbb1e622bbc8e8204d0f52a4e6614e06b5b9845d56c549cae1720073d7742"}
Jul 14 15:14:54.366454 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:54.366430    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879943728"
Jul 14 15:14:54.391215 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:54.391186    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7489-140672879943728_479d44a6-26ec-412f-ab83-8d558457f65d/test-curl-7489-140672879943728/0.log"
Jul 14 15:14:54.422764 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:54.422732    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7489-140672879943728"]
Jul 14 15:14:54.427607 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:54.427582    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7489-140672879943728"]
Jul 14 15:14:54.477996 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:54.477911    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtmf9\" (UniqueName: \"kubernetes.io/projected/479d44a6-26ec-412f-ab83-8d558457f65d-kube-api-access-mtmf9\") pod \"479d44a6-26ec-412f-ab83-8d558457f65d\" (UID: \"479d44a6-26ec-412f-ab83-8d558457f65d\") "
Jul 14 15:14:54.480116 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:54.480091    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479d44a6-26ec-412f-ab83-8d558457f65d-kube-api-access-mtmf9" (OuterVolumeSpecName: "kube-api-access-mtmf9") pod "479d44a6-26ec-412f-ab83-8d558457f65d" (UID: "479d44a6-26ec-412f-ab83-8d558457f65d"). InnerVolumeSpecName "kube-api-access-mtmf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:14:54.579786 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:54.579754    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtmf9\" (UniqueName: \"kubernetes.io/projected/479d44a6-26ec-412f-ab83-8d558457f65d-kube-api-access-mtmf9\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:14:54.876574 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:54.876495    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="479d44a6-26ec-412f-ab83-8d558457f65d" path="/var/lib/kubelet/pods/479d44a6-26ec-412f-ab83-8d558457f65d/volumes"
Jul 14 15:14:55.226096 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:55.226068    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7489-140672879943728"
Jul 14 15:14:55.226096 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:14:55.226095    2586 scope.go:117] "RemoveContainer" containerID="cd2fbb1e622bbc8e8204d0f52a4e6614e06b5b9845d56c549cae1720073d7742"
Jul 14 15:15:11.369621 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:11.369584    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"]
Jul 14 15:15:11.370225 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:11.369865    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv" podUID="383bc700-f214-41ce-94d6-41b624636719" containerName="maas-api" containerID="cri-o://2ca9da949db93d634ef8de721e9b4db2d628456827dfb8c7819a5f71d9fadc75" gracePeriod=30
Jul 14 15:15:11.829192 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:11.829165    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:15:11.845641 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:11.845612    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c42n7\" (UniqueName: \"kubernetes.io/projected/383bc700-f214-41ce-94d6-41b624636719-kube-api-access-c42n7\") pod \"383bc700-f214-41ce-94d6-41b624636719\" (UID: \"383bc700-f214-41ce-94d6-41b624636719\") "
Jul 14 15:15:11.845813 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:11.845653    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/383bc700-f214-41ce-94d6-41b624636719-openshift-service-ca\") pod \"383bc700-f214-41ce-94d6-41b624636719\" (UID: \"383bc700-f214-41ce-94d6-41b624636719\") "
Jul 14 15:15:11.845813 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:11.845675    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/383bc700-f214-41ce-94d6-41b624636719-maas-api-tls\") pod \"383bc700-f214-41ce-94d6-41b624636719\" (UID: \"383bc700-f214-41ce-94d6-41b624636719\") "
Jul 14 15:15:11.846012 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:11.845988    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383bc700-f214-41ce-94d6-41b624636719-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "383bc700-f214-41ce-94d6-41b624636719" (UID: "383bc700-f214-41ce-94d6-41b624636719"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:15:11.848330 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:11.848288    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383bc700-f214-41ce-94d6-41b624636719-kube-api-access-c42n7" (OuterVolumeSpecName: "kube-api-access-c42n7") pod "383bc700-f214-41ce-94d6-41b624636719" (UID: "383bc700-f214-41ce-94d6-41b624636719"). InnerVolumeSpecName "kube-api-access-c42n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:15:11.848491 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:11.848389    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383bc700-f214-41ce-94d6-41b624636719-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "383bc700-f214-41ce-94d6-41b624636719" (UID: "383bc700-f214-41ce-94d6-41b624636719"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:15:11.947175 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:11.947125    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c42n7\" (UniqueName: \"kubernetes.io/projected/383bc700-f214-41ce-94d6-41b624636719-kube-api-access-c42n7\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:15:11.947175 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:11.947170    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/383bc700-f214-41ce-94d6-41b624636719-openshift-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:15:11.947175 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:11.947180    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/383bc700-f214-41ce-94d6-41b624636719-maas-api-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:15:12.325386 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:12.325288    2586 generic.go:356] "Generic (PLEG): container finished" podID="383bc700-f214-41ce-94d6-41b624636719" containerID="2ca9da949db93d634ef8de721e9b4db2d628456827dfb8c7819a5f71d9fadc75" exitCode=0
Jul 14 15:15:12.325386 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:12.325343    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"
Jul 14 15:15:12.325660 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:12.325380    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv" event={"ID":"383bc700-f214-41ce-94d6-41b624636719","Type":"ContainerDied","Data":"2ca9da949db93d634ef8de721e9b4db2d628456827dfb8c7819a5f71d9fadc75"}
Jul 14 15:15:12.325660 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:12.325463    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv" event={"ID":"383bc700-f214-41ce-94d6-41b624636719","Type":"ContainerDied","Data":"79db80ddffa9c6c5f0d1d5d683936955c68dcda653023b7837cb450b93703ef2"}
Jul 14 15:15:12.325660 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:12.325494    2586 scope.go:117] "RemoveContainer" containerID="2ca9da949db93d634ef8de721e9b4db2d628456827dfb8c7819a5f71d9fadc75"
Jul 14 15:15:12.335162 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:12.335141    2586 scope.go:117] "RemoveContainer" containerID="2ca9da949db93d634ef8de721e9b4db2d628456827dfb8c7819a5f71d9fadc75"
Jul 14 15:15:12.335471 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:15:12.335453    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca9da949db93d634ef8de721e9b4db2d628456827dfb8c7819a5f71d9fadc75\": container with ID starting with 2ca9da949db93d634ef8de721e9b4db2d628456827dfb8c7819a5f71d9fadc75 not found: ID does not exist" containerID="2ca9da949db93d634ef8de721e9b4db2d628456827dfb8c7819a5f71d9fadc75"
Jul 14 15:15:12.335527 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:12.335480    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca9da949db93d634ef8de721e9b4db2d628456827dfb8c7819a5f71d9fadc75"} err="failed to get container status \"2ca9da949db93d634ef8de721e9b4db2d628456827dfb8c7819a5f71d9fadc75\": rpc error: code = NotFound desc = could not find container \"2ca9da949db93d634ef8de721e9b4db2d628456827dfb8c7819a5f71d9fadc75\": container with ID starting with 2ca9da949db93d634ef8de721e9b4db2d628456827dfb8c7819a5f71d9fadc75 not found: ID does not exist"
Jul 14 15:15:12.350026 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:12.349999    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"]
Jul 14 15:15:12.352847 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:12.352822    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ebb1a4-658f7ccb77-2w8vv"]
Jul 14 15:15:12.876059 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:12.876019    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="383bc700-f214-41ce-94d6-41b624636719" path="/var/lib/kubelet/pods/383bc700-f214-41ce-94d6-41b624636719/volumes"
Jul 14 15:15:23.051052 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:23.051017    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 15:15:23.057191 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:23.057170    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x8sbr_a61c2cbb-e090-4ca1-825b-e0b85151fcf2/ovn-acl-logging/0.log"
Jul 14 15:15:37.589301 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:37.589264    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"]
Jul 14 15:15:37.589791 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:37.589534    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z" podUID="5e436be0-6e51-4042-bb5f-8d77b15710b5" containerName="maas-api" containerID="cri-o://9e719d1208f5718e5f69ec17963fd2003fbc19a03951a1bf7ede66ae4b29f466" gracePeriod=30
Jul 14 15:15:38.349762 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.349732    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:15:38.466027 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.465991    2586 generic.go:356] "Generic (PLEG): container finished" podID="5e436be0-6e51-4042-bb5f-8d77b15710b5" containerID="9e719d1208f5718e5f69ec17963fd2003fbc19a03951a1bf7ede66ae4b29f466" exitCode=0
Jul 14 15:15:38.466225 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.466040    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"
Jul 14 15:15:38.466225 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.466079    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z" event={"ID":"5e436be0-6e51-4042-bb5f-8d77b15710b5","Type":"ContainerDied","Data":"9e719d1208f5718e5f69ec17963fd2003fbc19a03951a1bf7ede66ae4b29f466"}
Jul 14 15:15:38.466225 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.466134    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z" event={"ID":"5e436be0-6e51-4042-bb5f-8d77b15710b5","Type":"ContainerDied","Data":"bf74d8a304134956fb22c33e49d22b1bf8f611d88e687550bffba3fcc3c35a0d"}
Jul 14 15:15:38.466225 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.466156    2586 scope.go:117] "RemoveContainer" containerID="9e719d1208f5718e5f69ec17963fd2003fbc19a03951a1bf7ede66ae4b29f466"
Jul 14 15:15:38.475959 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.475942    2586 scope.go:117] "RemoveContainer" containerID="9e719d1208f5718e5f69ec17963fd2003fbc19a03951a1bf7ede66ae4b29f466"
Jul 14 15:15:38.476227 ip-10-0-141-209 kubenswrapper[2586]: E0714 15:15:38.476206    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e719d1208f5718e5f69ec17963fd2003fbc19a03951a1bf7ede66ae4b29f466\": container with ID starting with 9e719d1208f5718e5f69ec17963fd2003fbc19a03951a1bf7ede66ae4b29f466 not found: ID does not exist" containerID="9e719d1208f5718e5f69ec17963fd2003fbc19a03951a1bf7ede66ae4b29f466"
Jul 14 15:15:38.476266 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.476237    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e719d1208f5718e5f69ec17963fd2003fbc19a03951a1bf7ede66ae4b29f466"} err="failed to get container status \"9e719d1208f5718e5f69ec17963fd2003fbc19a03951a1bf7ede66ae4b29f466\": rpc error: code = NotFound desc = could not find container \"9e719d1208f5718e5f69ec17963fd2003fbc19a03951a1bf7ede66ae4b29f466\": container with ID starting with 9e719d1208f5718e5f69ec17963fd2003fbc19a03951a1bf7ede66ae4b29f466 not found: ID does not exist"
Jul 14 15:15:38.511625 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.511585    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e436be0-6e51-4042-bb5f-8d77b15710b5-maas-api-tls\") pod \"5e436be0-6e51-4042-bb5f-8d77b15710b5\" (UID: \"5e436be0-6e51-4042-bb5f-8d77b15710b5\") "
Jul 14 15:15:38.511808 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.511740    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbbps\" (UniqueName: \"kubernetes.io/projected/5e436be0-6e51-4042-bb5f-8d77b15710b5-kube-api-access-pbbps\") pod \"5e436be0-6e51-4042-bb5f-8d77b15710b5\" (UID: \"5e436be0-6e51-4042-bb5f-8d77b15710b5\") "
Jul 14 15:15:38.511808 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.511763    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e436be0-6e51-4042-bb5f-8d77b15710b5-openshift-service-ca\") pod \"5e436be0-6e51-4042-bb5f-8d77b15710b5\" (UID: \"5e436be0-6e51-4042-bb5f-8d77b15710b5\") "
Jul 14 15:15:38.512169 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.512140    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e436be0-6e51-4042-bb5f-8d77b15710b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5e436be0-6e51-4042-bb5f-8d77b15710b5" (UID: "5e436be0-6e51-4042-bb5f-8d77b15710b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 15:15:38.514098 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.514076    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e436be0-6e51-4042-bb5f-8d77b15710b5-kube-api-access-pbbps" (OuterVolumeSpecName: "kube-api-access-pbbps") pod "5e436be0-6e51-4042-bb5f-8d77b15710b5" (UID: "5e436be0-6e51-4042-bb5f-8d77b15710b5"). InnerVolumeSpecName "kube-api-access-pbbps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 15:15:38.514182 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.514138    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e436be0-6e51-4042-bb5f-8d77b15710b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5e436be0-6e51-4042-bb5f-8d77b15710b5" (UID: "5e436be0-6e51-4042-bb5f-8d77b15710b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 15:15:38.612533 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.612496    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e436be0-6e51-4042-bb5f-8d77b15710b5-maas-api-tls\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:15:38.612533 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.612525    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbbps\" (UniqueName: \"kubernetes.io/projected/5e436be0-6e51-4042-bb5f-8d77b15710b5-kube-api-access-pbbps\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:15:38.612533 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.612537    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e436be0-6e51-4042-bb5f-8d77b15710b5-openshift-service-ca\") on node \"ip-10-0-141-209.ec2.internal\" DevicePath \"\""
Jul 14 15:15:38.792166 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.792131    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"]
Jul 14 15:15:38.795759 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.795729    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-747ac7-8544db68d8-mck5z"]
Jul 14 15:15:38.880032 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:38.879999    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e436be0-6e51-4042-bb5f-8d77b15710b5" path="/var/lib/kubelet/pods/5e436be0-6e51-4042-bb5f-8d77b15710b5/volumes"
Jul 14 15:15:53.745164 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:53.745121    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b68b66c55-74sdf_5d3edc5b-855e-46ba-9585-55ee5665fa97/authorino/0.log"
Jul 14 15:15:59.470146 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:59.470107    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5898bfff6b-r5frs_fbd9157b-90f9-4527-afc4-56feb96a2268/manager/0.log"
Jul 14 15:15:59.840646 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:59.840573    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4469fb8c-8qbxj_cf559f9d-f954-4dd8-bb53-718ecb7640fb/copy-manifests/0.log"
Jul 14 15:15:59.847684 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:15:59.847636    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4469fb8c-8qbxj_cf559f9d-f954-4dd8-bb53-718ecb7640fb/manager/0.log"
Jul 14 15:16:00.505893 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:00.505855    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5b4d9fd998-qx6v9_aa1e3107-9226-4b8d-8d2c-a30a94c1a6e3/maas-api/0.log"
Jul 14 15:16:00.629746 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:00.629718    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734005-bwdwp_6af5f455-5199-41c1-9f91-349ef3898d0c/cleanup/0.log"
Jul 14 15:16:00.745856 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:00.745820    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734020-54d2m_971bdfa3-4000-431a-bc2a-a88cb66599ee/cleanup/0.log"
Jul 14 15:16:00.983608 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:00.983575    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-2b2b03f7-rxfkw_537ca958-8074-4453-86c5-b2f94eb839bc/revoke-keys/0.log"
Jul 14 15:16:01.102446 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:01.102400    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-9cfd4b9b-c487j_25e734f3-56eb-4ebb-91b7-851263fc0e33/revoke-keys/0.log"
Jul 14 15:16:01.340568 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:01.340474    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-40b1bd-5bptl_e242f140-3f9b-4446-8b0a-e7dbfc455d20/revoke-keys/0.log"
Jul 14 15:16:01.464777 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:01.464743    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-4b164646-bxpj8_761ce6c7-face-4c27-95c0-fd7af330cb84/revoke-keys/0.log"
Jul 14 15:16:01.697933 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:01.697906    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-4226a2-bwktx_0ae57172-c7bf-4384-9926-095ff55b3105/revoke-keys/0.log"
Jul 14 15:16:01.842780 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:01.842743    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-11424108-sl42n_66c0b8bd-3911-482f-a28a-4b1ae46d7833/revoke-keys/0.log"
Jul 14 15:16:02.110793 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:02.110699    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-beb2a668-q77r5_e0238dda-d3c0-4c7f-8b6f-6c2aa4acb560/revoke-keys/0.log"
Jul 14 15:16:03.988362 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:03.988329    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b68b66c55-74sdf_5d3edc5b-855e-46ba-9585-55ee5665fa97/authorino/0.log"
Jul 14 15:16:04.645635 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:04.645596    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dx7fp_318d09e6-2ac2-42e1-beb2-99f961d7e886/manager/0.log"
Jul 14 15:16:05.700854 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:05.700826    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f65956896-97cm7_e1782e59-c729-4ce9-a67e-d5f5a9950ad0/kube-auth-proxy/0.log"
Jul 14 15:16:06.261947 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:06.261905    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5965b849d4-jl74t_8bbbac86-f54f-4df4-9839-ecbc28741a9f/router/0.log"
Jul 14 15:16:06.633739 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:06.633648    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-9wxgx_d661f6cc-1b4e-4ba5-980c-911e406264bd/main/0.log"
Jul 14 15:16:06.900481 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:06.900376    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-d4fk6_5117094c-0aa4-4043-a8a7-88cf03e92a64/main/0.log"
Jul 14 15:16:11.475687 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.475654    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x95g4/must-gather-tw8sc"]
Jul 14 15:16:11.476151 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.476046    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="479d44a6-26ec-412f-ab83-8d558457f65d" containerName="test-curl-7489-140672879943728"
Jul 14 15:16:11.476151 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.476061    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="479d44a6-26ec-412f-ab83-8d558457f65d" containerName="test-curl-7489-140672879943728"
Jul 14 15:16:11.476151 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.476078    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383bc700-f214-41ce-94d6-41b624636719" containerName="maas-api"
Jul 14 15:16:11.476151 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.476084    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="383bc700-f214-41ce-94d6-41b624636719" containerName="maas-api"
Jul 14 15:16:11.476151 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.476092    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e436be0-6e51-4042-bb5f-8d77b15710b5" containerName="maas-api"
Jul 14 15:16:11.476151 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.476098    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e436be0-6e51-4042-bb5f-8d77b15710b5" containerName="maas-api"
Jul 14 15:16:11.476151 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.476154    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e436be0-6e51-4042-bb5f-8d77b15710b5" containerName="maas-api"
Jul 14 15:16:11.476382 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.476165    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="383bc700-f214-41ce-94d6-41b624636719" containerName="maas-api"
Jul 14 15:16:11.476382 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.476173    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="479d44a6-26ec-412f-ab83-8d558457f65d" containerName="test-curl-7489-140672879943728"
Jul 14 15:16:11.479041 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.479014    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x95g4/must-gather-tw8sc"
Jul 14 15:16:11.482440 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.482391    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x95g4\"/\"default-dockercfg-czkgw\""
Jul 14 15:16:11.482440 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.482405    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x95g4\"/\"kube-root-ca.crt\""
Jul 14 15:16:11.484722 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.484697    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x95g4\"/\"openshift-service-ca.crt\""
Jul 14 15:16:11.496576 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.496547    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x95g4/must-gather-tw8sc"]
Jul 14 15:16:11.616819 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.616778    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f-must-gather-output\") pod \"must-gather-tw8sc\" (UID: \"f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f\") " pod="openshift-must-gather-x95g4/must-gather-tw8sc"
Jul 14 15:16:11.617019 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.616885    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkqrd\" (UniqueName: \"kubernetes.io/projected/f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f-kube-api-access-nkqrd\") pod \"must-gather-tw8sc\" (UID: \"f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f\") " pod="openshift-must-gather-x95g4/must-gather-tw8sc"
Jul 14 15:16:11.717878 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.717836    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkqrd\" (UniqueName: \"kubernetes.io/projected/f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f-kube-api-access-nkqrd\") pod \"must-gather-tw8sc\" (UID: \"f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f\") " pod="openshift-must-gather-x95g4/must-gather-tw8sc"
Jul 14 15:16:11.718055 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.717904    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f-must-gather-output\") pod \"must-gather-tw8sc\" (UID: \"f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f\") " pod="openshift-must-gather-x95g4/must-gather-tw8sc"
Jul 14 15:16:11.718256 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.718239    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f-must-gather-output\") pod \"must-gather-tw8sc\" (UID: \"f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f\") " pod="openshift-must-gather-x95g4/must-gather-tw8sc"
Jul 14 15:16:11.740058 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.739986    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkqrd\" (UniqueName: \"kubernetes.io/projected/f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f-kube-api-access-nkqrd\") pod \"must-gather-tw8sc\" (UID: \"f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f\") " pod="openshift-must-gather-x95g4/must-gather-tw8sc"
Jul 14 15:16:11.789160 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.789126    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x95g4/must-gather-tw8sc"
Jul 14 15:16:11.938760 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:11.938737    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x95g4/must-gather-tw8sc"]
Jul 14 15:16:11.940779 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:16:11.940751    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf41aaeb5_ebc6_4375_ab05_dc1fd261ef9f.slice/crio-e60b0d562b2018b001f30e968d7e2158e143af0b278a2d521d8f5e29a6dfde57 WatchSource:0}: Error finding container e60b0d562b2018b001f30e968d7e2158e143af0b278a2d521d8f5e29a6dfde57: Status 404 returned error can't find the container with id e60b0d562b2018b001f30e968d7e2158e143af0b278a2d521d8f5e29a6dfde57
Jul 14 15:16:12.657261 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:12.657219    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x95g4/must-gather-tw8sc" event={"ID":"f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f","Type":"ContainerStarted","Data":"e60b0d562b2018b001f30e968d7e2158e143af0b278a2d521d8f5e29a6dfde57"}
Jul 14 15:16:13.667356 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:13.667314    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x95g4/must-gather-tw8sc" event={"ID":"f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f","Type":"ContainerStarted","Data":"9be82b8e3c6e6f91779648cf2a63e1151935db47ae76339c6c1d0268f98e5def"}
Jul 14 15:16:13.667890 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:13.667367    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x95g4/must-gather-tw8sc" event={"ID":"f41aaeb5-ebc6-4375-ab05-dc1fd261ef9f","Type":"ContainerStarted","Data":"cf1be62c2e12695e61c5becfe68e629fe6a8615aa584a5386c467e1bd95e80d3"}
Jul 14 15:16:13.699720 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:13.699652    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x95g4/must-gather-tw8sc" podStartSLOduration=1.7639583810000001 podStartE2EDuration="2.699629901s" totalImagesPullingTime="935.67152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 15:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:16:13.698130355 +0000 UTC m=+2751.364316776" watchObservedRunningTime="2026-07-14 15:16:13.699629901 +0000 UTC m=+2751.365816323"
Jul 14 15:16:14.753686 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:14.753644    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rlbbv_3febdf7d-4462-4194-a512-7c860fc44797/global-pull-secret-syncer/0.log"
Jul 14 15:16:15.040707 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:15.040626    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t84p9_f12e6b3b-d351-4faf-bf16-9ccfd4100848/konnectivity-agent/0.log"
Jul 14 15:16:15.147570 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:15.147545    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-209.ec2.internal_32e6ac82d25811564f3e9e457bd986bc/haproxy/0.log"
Jul 14 15:16:20.624527 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:20.624490    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b68b66c55-74sdf_5d3edc5b-855e-46ba-9585-55ee5665fa97/authorino/0.log"
Jul 14 15:16:20.986440 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:20.982182    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dx7fp_318d09e6-2ac2-42e1-beb2-99f961d7e886/manager/0.log"
Jul 14 15:16:23.037393 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.037360    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-sgrwz_a26ee4e3-b7fb-4ed4-9056-ae52a53b8667/cluster-monitoring-operator/0.log"
Jul 14 15:16:23.072580 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.072550    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-56qqr_6e84e3d2-b8b1-48e4-81a5-6d5d28d42898/kube-state-metrics/0.log"
Jul 14 15:16:23.095658 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.095566    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-56qqr_6e84e3d2-b8b1-48e4-81a5-6d5d28d42898/kube-rbac-proxy-main/0.log"
Jul 14 15:16:23.127124 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.127034    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-56qqr_6e84e3d2-b8b1-48e4-81a5-6d5d28d42898/kube-rbac-proxy-self/0.log"
Jul 14 15:16:23.422180 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.422105    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mfkkg_d50dbe94-4da7-4160-ab95-7f028a88d5b6/node-exporter/0.log"
Jul 14 15:16:23.449720 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.449675    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mfkkg_d50dbe94-4da7-4160-ab95-7f028a88d5b6/kube-rbac-proxy/0.log"
Jul 14 15:16:23.471924 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.471883    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mfkkg_d50dbe94-4da7-4160-ab95-7f028a88d5b6/init-textfile/0.log"
Jul 14 15:16:23.514090 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.514058    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-vjcnd_41817d03-f7be-4b25-991e-7deaaf44b9dc/kube-rbac-proxy-main/0.log"
Jul 14 15:16:23.539107 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.539017    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-vjcnd_41817d03-f7be-4b25-991e-7deaaf44b9dc/kube-rbac-proxy-self/0.log"
Jul 14 15:16:23.562614 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.562586    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-vjcnd_41817d03-f7be-4b25-991e-7deaaf44b9dc/openshift-state-metrics/0.log"
Jul 14 15:16:23.601809 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.601772    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_56c8841f-9fb2-4ece-a4fb-06c5f159491e/prometheus/0.log"
Jul 14 15:16:23.631180 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.631154    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_56c8841f-9fb2-4ece-a4fb-06c5f159491e/config-reloader/0.log"
Jul 14 15:16:23.662942 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.662919    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_56c8841f-9fb2-4ece-a4fb-06c5f159491e/thanos-sidecar/0.log"
Jul 14 15:16:23.687893 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.687864    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_56c8841f-9fb2-4ece-a4fb-06c5f159491e/kube-rbac-proxy-web/0.log"
Jul 14 15:16:23.712536 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.712505    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_56c8841f-9fb2-4ece-a4fb-06c5f159491e/kube-rbac-proxy/0.log"
Jul 14 15:16:23.746891 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.746856    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_56c8841f-9fb2-4ece-a4fb-06c5f159491e/kube-rbac-proxy-thanos/0.log"
Jul 14 15:16:23.772871 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.772840    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_56c8841f-9fb2-4ece-a4fb-06c5f159491e/init-config-reloader/0.log"
Jul 14 15:16:23.805615 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.805577    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-z6lcx_16cee93e-ca72-44e1-a79b-670955bea80b/prometheus-operator/0.log"
Jul 14 15:16:23.831689 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.831661    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-z6lcx_16cee93e-ca72-44e1-a79b-670955bea80b/kube-rbac-proxy/0.log"
Jul 14 15:16:23.862202 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.862160    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-bqmmx_29a1fc9f-e7cf-46f6-879d-957d1a46253c/prometheus-operator-admission-webhook/0.log"
Jul 14 15:16:23.894698 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.894656    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6df7fcbc56-ptd2m_f3e6b198-b9d4-48d2-a3c7-7c4528352d8f/telemeter-client/0.log"
Jul 14 15:16:23.918738 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.918700    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6df7fcbc56-ptd2m_f3e6b198-b9d4-48d2-a3c7-7c4528352d8f/reload/0.log"
Jul 14 15:16:23.944197 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.944121    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6df7fcbc56-ptd2m_f3e6b198-b9d4-48d2-a3c7-7c4528352d8f/kube-rbac-proxy/0.log"
Jul 14 15:16:23.975803 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:23.975777    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797bb6c9fc-4zqqz_698fc65b-ac4e-4a74-847d-f23611312764/thanos-query/0.log"
Jul 14 15:16:24.000598 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:24.000568    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797bb6c9fc-4zqqz_698fc65b-ac4e-4a74-847d-f23611312764/kube-rbac-proxy-web/0.log"
Jul 14 15:16:24.027728 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:24.027695    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797bb6c9fc-4zqqz_698fc65b-ac4e-4a74-847d-f23611312764/kube-rbac-proxy/0.log"
Jul 14 15:16:24.052502 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:24.052467    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797bb6c9fc-4zqqz_698fc65b-ac4e-4a74-847d-f23611312764/prom-label-proxy/0.log"
Jul 14 15:16:24.076108 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:24.076076    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797bb6c9fc-4zqqz_698fc65b-ac4e-4a74-847d-f23611312764/kube-rbac-proxy-rules/0.log"
Jul 14 15:16:24.100775 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:24.100741    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797bb6c9fc-4zqqz_698fc65b-ac4e-4a74-847d-f23611312764/kube-rbac-proxy-metrics/0.log"
Jul 14 15:16:25.187490 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.187445    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"]
Jul 14 15:16:25.192721 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.192688    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.201289 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.200745    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"]
Jul 14 15:16:25.362795 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.362761    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-tprcb_745b2f3b-5fd7-4ca7-885d-c5a465c91a6f/networking-console-plugin/0.log"
Jul 14 15:16:25.363160 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.363133    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/56f838d7-e2f8-422f-ba16-52d64225202e-proc\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.363287 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.363231    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56f838d7-e2f8-422f-ba16-52d64225202e-sys\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.363337 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.363297    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56f838d7-e2f8-422f-ba16-52d64225202e-lib-modules\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.363389 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.363342    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/56f838d7-e2f8-422f-ba16-52d64225202e-podres\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.363464 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.363439    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8twb\" (UniqueName: \"kubernetes.io/projected/56f838d7-e2f8-422f-ba16-52d64225202e-kube-api-access-g8twb\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.464440 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.464318    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56f838d7-e2f8-422f-ba16-52d64225202e-lib-modules\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.464714 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.464689    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/56f838d7-e2f8-422f-ba16-52d64225202e-podres\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.464857 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.464830    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/56f838d7-e2f8-422f-ba16-52d64225202e-podres\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.465014 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.464913    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8twb\" (UniqueName: \"kubernetes.io/projected/56f838d7-e2f8-422f-ba16-52d64225202e-kube-api-access-g8twb\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.465153 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.465137    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/56f838d7-e2f8-422f-ba16-52d64225202e-proc\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.465290 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.465275    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56f838d7-e2f8-422f-ba16-52d64225202e-sys\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.465433 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.465348    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/56f838d7-e2f8-422f-ba16-52d64225202e-proc\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.465560 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.464550    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56f838d7-e2f8-422f-ba16-52d64225202e-lib-modules\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.465560 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.465498    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56f838d7-e2f8-422f-ba16-52d64225202e-sys\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.475659 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.475634    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8twb\" (UniqueName: \"kubernetes.io/projected/56f838d7-e2f8-422f-ba16-52d64225202e-kube-api-access-g8twb\") pod \"perf-node-gather-daemonset-6xxhw\" (UID: \"56f838d7-e2f8-422f-ba16-52d64225202e\") " pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.506194 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.506162    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:25.675599 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.675568    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"]
Jul 14 15:16:25.677060 ip-10-0-141-209 kubenswrapper[2586]: W0714 15:16:25.677029    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod56f838d7_e2f8_422f_ba16_52d64225202e.slice/crio-6ad5ba6c35dcd2fae924bdb31055249e4f957ec5a010a259094e9db9f8890619 WatchSource:0}: Error finding container 6ad5ba6c35dcd2fae924bdb31055249e4f957ec5a010a259094e9db9f8890619: Status 404 returned error can't find the container with id 6ad5ba6c35dcd2fae924bdb31055249e4f957ec5a010a259094e9db9f8890619
Jul 14 15:16:25.759129 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:25.759073    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw" event={"ID":"56f838d7-e2f8-422f-ba16-52d64225202e","Type":"ContainerStarted","Data":"6ad5ba6c35dcd2fae924bdb31055249e4f957ec5a010a259094e9db9f8890619"}
Jul 14 15:16:26.766648 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:26.766600    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw" event={"ID":"56f838d7-e2f8-422f-ba16-52d64225202e","Type":"ContainerStarted","Data":"5fcdfb88f5a3bdedd0aedbe166c30175718984c7a97eb51eb258f4d7e1a60b00"}
Jul 14 15:16:26.767222 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:26.766665    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:26.788726 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:26.788668    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw" podStartSLOduration=1.788653623 podStartE2EDuration="1.788653623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 15:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 15:16:26.785639034 +0000 UTC m=+2764.451825455" watchObservedRunningTime="2026-07-14 15:16:26.788653623 +0000 UTC m=+2764.454840043"
Jul 14 15:16:28.237781 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:28.237746    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2f5v2_dd676278-6891-49d4-881e-41f5b8029e62/dns/0.log"
Jul 14 15:16:28.260609 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:28.260572    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2f5v2_dd676278-6891-49d4-881e-41f5b8029e62/kube-rbac-proxy/0.log"
Jul 14 15:16:28.402546 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:28.402503    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fl6rb_401faa4d-42da-4c1c-af1c-fe150089dfe0/dns-node-resolver/0.log"
Jul 14 15:16:29.125165 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:29.125133    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nqtwh_74295487-4d63-42cb-b367-fda447a5217b/node-ca/0.log"
Jul 14 15:16:30.346516 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:30.346481    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f65956896-97cm7_e1782e59-c729-4ce9-a67e-d5f5a9950ad0/kube-auth-proxy/0.log"
Jul 14 15:16:30.544997 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:30.544962    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5965b849d4-jl74t_8bbbac86-f54f-4df4-9839-ecbc28741a9f/router/0.log"
Jul 14 15:16:31.326094 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:31.326037    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wxlrn_8b115d5b-3bdb-4b02-b747-a749b79536e4/serve-healthcheck-canary/0.log"
Jul 14 15:16:31.817277 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:31.817243    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-hqvkg_a2e3a1f9-cde2-4a79-ae33-9467155e2f14/insights-operator/0.log"
Jul 14 15:16:31.818796 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:31.818769    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-hqvkg_a2e3a1f9-cde2-4a79-ae33-9467155e2f14/insights-operator/1.log"
Jul 14 15:16:31.843850 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:31.843819    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5sn8c_2e0ef85f-65de-4d78-873a-9ce585c997cf/kube-rbac-proxy/0.log"
Jul 14 15:16:31.869461 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:31.869429    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5sn8c_2e0ef85f-65de-4d78-873a-9ce585c997cf/exporter/0.log"
Jul 14 15:16:31.897800 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:31.897771    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5sn8c_2e0ef85f-65de-4d78-873a-9ce585c997cf/extractor/0.log"
Jul 14 15:16:32.784050 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:32.784025    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x95g4/perf-node-gather-daemonset-6xxhw"
Jul 14 15:16:34.904997 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:34.904927    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5898bfff6b-r5frs_fbd9157b-90f9-4527-afc4-56feb96a2268/manager/0.log"
Jul 14 15:16:35.068115 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:35.068086    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4469fb8c-8qbxj_cf559f9d-f954-4dd8-bb53-718ecb7640fb/manager/0.log"
Jul 14 15:16:35.092357 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:35.092321    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b4469fb8c-8qbxj_cf559f9d-f954-4dd8-bb53-718ecb7640fb/copy-manifests/0.log"
Jul 14 15:16:35.815850 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:35.815821    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f65956896-97cm7_e1782e59-c729-4ce9-a67e-d5f5a9950ad0/kube-auth-proxy/0.log"
Jul 14 15:16:35.981752 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:35.981726    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5965b849d4-jl74t_8bbbac86-f54f-4df4-9839-ecbc28741a9f/router/0.log"
Jul 14 15:16:36.529557 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:36.529482    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-fc6b4b856-v6ghr_5e1ad1f2-2d10-4009-a3e9-307393af3049/manager/0.log"
Jul 14 15:16:37.525667 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:37.525588    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f65956896-97cm7_e1782e59-c729-4ce9-a67e-d5f5a9950ad0/kube-auth-proxy/0.log"
Jul 14 15:16:37.707745 ip-10-0-141-209 kubenswrapper[2586]: I0714 15:16:37.707718    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5965b849d4-jl74t_8bbbac86-f54f-4df4-9839-ecbc28741a9f/router/0.log"